A personal blog about some hobbies of mine.
Date: June 24, 2020
Today’s accomplishment was crafting the first cut of a Step Function deployed via SAM & CloudFormation. I went head first into writing code… and quickly realized my previous drawing needed some more love. I redrew my previous step function so I could track the input parameters and detail the decision points. Here is that new drawing along with how AWS visualized it via the Step Functions console:
Date: June 23, 2020
Part of today was spent at Mount Rainier (Link: Photos in iCloud), so I did not put in a full day’s effort. Today’s updates (Link: Commits in GitHub) involve parsing the response from Athena’s get_query_results() method. It is not pretty but it does the job:
Date: June 22, 2020
I said I was not going to work on this project over the weekend. That was a lie: I pushed a few commits into the repo through the weekend. Last I wrote, I was a bit frustrated by the response structure provided by Athena’s get_query_results() method. This response is a row-based dictionary where each element lists the related columns’ values. It is probably the simplest way to share tabular results that do not have a primary key but it flies in the face against column-oriented data types that have become a modern standard.
Date: June 19, 2020
You see this image below? It scares me. This is how a query looks when you ask Athena to grab query results for you. It mimics the rows and columns in terms of how a person would think of a query result. The raw JSON file is available at the end of this post. Let’s take a step back and talk about how I got here.
Date: June 18, 2020
At this point, all of the infrastructure work is complete and I am pulling stock data every minute for 11 stocks. The biggest additions from yesterday include using a time-triggered Lambda to queue up stock data requests for another Lambda to go get. The results get moved into a data stream then stored in a data lake. Now I have a data catalog available that enables us to query Amazon Athena (serverless query service) to do some basic analytics on the data. The latest hand-drawn monstrosity of an architecture diagram looks like this:
Date: June 17, 2020
The basic infrastructure is complete: code grabs stock data from Finnhub and pushes the result into a data stream where it eventually gets stored in a data lake. The API secret token is stored in AWS Secrets Manager and never exposed. Everything is deployed via CloudFormation; you can start looking at my code on Github.