JSON

Following on from this post, I thought it might be good to think about JavaScript Object Notation (JSON).

Despite its name, JSON has evolved to be agnostic and is arguably the standard for exchanging information because its human readable and easy for computers to parse.

JSON is two things: a collection of key value pairs, and an ordered list of values. The former is easily represented with objects, dictionaries, etc. and he latter with arrays, lists, or sequences.

  • Values are string, number, object, array, true/false, or null surrounded by white-space.
  • Objects are unordered key-value pairs, represented by colon separated key-value pairs and wrapped in curly brackets.
  • Arrays are comma separated ordered collections wrapped in square brackets.
  • Comments, trailing commas, and single quoted strings are not supported.
{
    "name" : "Dougie",
    "languages" : [ "python", "java" ]
}

Python has a standard module, json, making it simple to convert Python and JSON data types. Serialisation is converting to JSON, but take care with de-serialisation as it is not quite the reverse. Not all Python data types can be converted to JSON values, so objects may not retain their original type.

This can introduce subtle problems when serialising and de-serialising:

  • JSON requires keys to be strings, so where another type is given during serialisation, it will be a string when de-serialised.
  • True serialises to true, and None serialises to null. The reverse is also true.
  • Tuple are serialised as arrays, there is no way to tell when de-serialising so you will get a list.

Serialising is done by dumping:

json.dumps(python_data)

Deserialising JSON is done by loading:

json.loads(json_data)

JSON is human readable but can be made easier to read by specifying indentation level:

json.dumps(sample_json, indent=2)

Interact with API in Python

Overview of REST

Representational State Transfer (REST) is a client server communication pattern, which defines some constraints:

  • Stateless
  • Decouple server from client
  • Should be cacheable.
  • Uniform interface
  • Layered (access directly or indirectly via proxy or load balancer).

Any service that adhere to this is known as a REST web service and provides access via REST API, which listens for HTTP methods and responds with an HTTP response. These API expose public URI, or endpoints, performing different actions dependent on the method.

Python and REST

I’ll do a separate post on writing API, because its more complicated and there are several frameworks (Flask and FastAPI for example), today we’ll focus on consuming API.

Pre-requisites

We’ll be using the requests library. As usual we’ll create a virtual environment:

python -m venv rest_env
source vest_env/bin/activate
python -m pip install requests

Now we need an API to consume, so lets use NASA’s public API. Discovering API isn’t something we can easily do programmatically, so we’re reliant on the API’s documentation. It’s also likely that responses will be JavaScript Object Notation (JSON) and we’ll consider how we parse that data a bit later on.

We’re always going to need import requests and specify a base URL, we’ll use Astronomy Picture of the Day (APOD):

import requests
api_url = "https://api.nasa.gov/planetary/apod"

Now we create a response object depending on the HTTP method:

response = requests.get(api_url)

We want to check the response:

>>> response.status_code
403

But wait, we want 200 (success) and we got 403 (forbidden) – the API requires authentication parameters!

Query string

This is the part of the URL that assigns values to a parameter. The APOD API requires a key as a parameter, you can request one here but I’ll use the demo key here. We’ll also request high definition images while we’re at it:

api_key = 'DEMO_KEY'
params = {
    'api_key':api_key,
    'hd':'True'
}
response.get(api_url, params=params)

Now when we check the status:

>>> response.status_code
200

Success!

Parsing the response

How do we get a specific attribute? Our response is in JSON. We can make this easier to work with by converting the response to a dictionary:

>>> import json
>>> response_dict = response.json()
>>> print(json.dumps(response_dict, indent=4, sort_keys=True))
{
    "date": "2025-09-14",
    "explanation": "How does your favorite planet spin? Does it spin rapidly around a nearly vertical axis, or horizontally, or backwards?  The featured video animates NASA images of all eight planets in our Solar System to show them spinning side-by-side for an easy comparison. In the time-lapse video, a day on Earth -- one Earth rotation -- takes just a few seconds.  Jupiter rotates the fastest, while Venus spins not only the slowest (can you see it?), but backwards.  The inner rocky planets across the top underwent dramatic spin-altering collisions during the early days of the Solar System.  Why planets spin and tilt as they do remains a topic of research with much insight gained from modern computer modeling and the recent discovery and analysis of hundreds of exoplanets: planets orbiting other stars.",
    "media_type": "video",
    "service_version": "v1",
    "title": "Planets of the Solar System: Tilts and Spins",
    "url": "https://www.youtube.com/embed/my1euFQHH-o?rel=0"
}

Now that we’ve a dictionary, getting the required attribute sis trivial:

>>> print(response_dict['url'])
https://www.youtube.com/embed/my1euFQHH-o?rel=0

POST and PUT

POST requests a new resource but PUT replaces a resource and creates it if it doesn’t exist, in other words PUT is idempotent where as POST is not.

JSON is also how we provide payloads. Lets take a look at an example that expects a product name and a quantity:

payload={"product_name": "Milk", "quantity": 1}
response = requests.put(api_url, json=payload)

Next time

We’ll follow this post up with a look at authentication, as most API will require some form of authentication.

Multipass cloud-init

Multipass is pretty useful but what a pain this was to figure out, due to Ubuntu’s Node.js package not working with AWS-CDK.

Multipass lets you manage VM in Ubuntu and can take cloud-init scripts as a parameter. I wanted an Ubuntu LTS instance with AWS CDK, which needs Node.js and python3-venv.

#cloud-config
packages:
  - python3-venv
  - unzip

package_update: true

package_upgrade: true

write_files:
  - path: "/etc/environment"
    append: true
    content: |
      export PATH=\
      /opt/node-v20.11.1-linux-x64/bin:\
      /usr/local/sbin:/usr/local/bin:\
      /usr/sbin:/usr/bin:/sbin:/bin:\
      /usr/games:/usr/local/games:\
      /snap/bin

runcmd:
  - wget https://nodejs.org/dist/v20.11.1/node-v20.11.1-linux-x64.tar.xz 
  - tar xvf node-v20.11.1-linux-x64.tar.xz -C /opt
  - export PATH=/opt/node-v20.11.1-linux-x64/bin:$PATH
  - npm install -g npm@latest
  - npm install -g aws-cdk
  - git config --system user.name "Dougie Richardson"
  - git config --system user.email "xx@xxxxxxxxx.com"
  - git config --system init.defaultBranch main
  - wget https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip
  - unzip awscli-exe-linux-x86_64.zip
  - ./aws/install

Save that as cdk.yaml and spin up an new instance:

multipass launch --name cdk --cloud-init cdk.yaml
Success!

There’s a couple useful things to note if you’re checking this out:

  • Inside the VM there’s a useful log to assist debugging /var/log/cloud-init-output.log.
  • While YAML has lots of ways to split text over multiple lines, when you don’t want space use a backslash.

Shell into the new VM with multipass shell cdk, then we can configure programmatic access and bootstrap CDK.

aws sso configure
aws sso login --profile profile_name
aws sts get-caller-identity --profile profile_name
aws configure get region --profile profile_name

The last two commands give the account and region to bootstrap:

cdk bootstrap aws://account_number/region --profile profile_name