GUI Design Follup Leads Me Down the Rabbit Hole
An Curious Adventure in the Wonderland of POST Requests and ModSecurity
So, yesterday I had set a task of putting together the GUI for the collect
and compute
routes. The GUI, as described in yesterday’s post, would make use of /api/v1/api_coverage
route that uses Flask’s internal application structure to track which functions have and have not been implemented. Simple enough.
Some preliminary testing reminded me that this route depends on the sspi_metadata
database, which contains useful information like the list of all the indicators and the country groups. In particular, the /api/v1/api_coverage
route uses the list of all indicator codes in sspi_metadata
to return a JSON object with the appropriate boolean value next to each indicator. Currently this powers a little progress tracker, but the same function could be used to implement the GUI.
The only problem was that the sspi_metadata
file had not been yet loaded onto the Linode, which meant that the /api/v1/api_coverage
route would not work until I got that data onto the server. In my local environment, I had written a janky little POST
route setup to receive data I was sending via the httr
library from R. The main issue with it, which I had run into a number of times over the course of the last couple months, was that it’s troublesome to get the session in R to hold onto the login cookie assigned by flask-login
long enough to POST
the data to the appropriate @login_required
route. This resulted in numerous headaches, so I’ve been avoiding hammering out these issues for a little while.
But, it seemed, today was the day. To successfully POST
to the server, I’d first first need to get through the HTTP
Basic Authentication that I setup with .htpasswd
to protect the site while in development. Unfortunately, I had forgotten that I’d put this protection in place and spent a few very confused minutes wondering why even my GET
requests were receiving a 401 error from R and Postman even though my browser was working just fine.
Turns out, I’d set my browser to remember my HTTP login credentials, which was convenient for me checking the site but proved very inconvenient indeed when it came time to try to deal with data. Well, only about fifteen minutes and a bunch of frustrated spamming down the drain. I’ll simply pass in the WWW-Authorize Basic Authorization header with my request and be on my way….
\403 Error. Sigh. Even with the appropriate authorization, the server is refusing to process the request. O…K? It’s around this time I finally wise up and check the actual error logs on the server. And, sure enough, ModSecurity
has flagged some admittedly suspicious traffic from my IP address.
I was sitting on a bench by South Hall, right across from the Campanile. It was getting chilly as the sun was starting to set, and I was completely frustrated, so I figured I’d walk to my office. On the way, I worked the problem in my head. I really didn’t want to fiddle with ModSecurity
: I configured it months ago and don’t really want to disable or weaken any protections. So then posting data in this way isn’t going to work. How to get the data onto the server then? Well, it’s all just in JSON files. Instead of posting them through the internet, I could just run a route the picks them up from the filesystem. After all, this is not an operation that needs to be run regularly, and storing a local JSON copy on the machine might not be the worst redundancy in the world.
And thus it was settled. It’s really amazing how changing your physical environment can completely shift your perspective on an abstract problem. A math professor of mine once advised us that most mathematicians have their greatest insights after stepping away from their desks for a walk after turning something over for a while. This definitely held true for me here.
So anyway, I got to my office and dashed out a few routes that would create a few buttons for loading and reloading static files into the appropriate databases. It works like this:
/api/v1
, the page which will house all the GUI controls for the backend, makes an AJAX call to theapi/v1/local
route under the heading “Local Data”/api/v1/local
calls a function to check for files in thelocal
directory and harvest their names. These are used to populate, render and return a Jinja2 template serverside equipped with the harvested names and buttons for each file in the directory, which are then slotted into the HTML for the main page when the AJAX request completes./api/v1/local/reload/<database>
is aPOST
route operated by each of the buttons. The appropriate<database>
argument gets dropped into theaction
field for each form when the template gets rendered serverside. When the button is clicked, the form is submitted, which activates the route. The input string is handled safely by a mapping function that fetches the appropriate database client, then all observations are dropped then immediately reloaded from the local file. The only thing left to do was toscp
over the copies of the JSON files from my machine to the Linode, which was easy enough. A few more minor snags having to do with the filesystem along the way notwithstanding, the basic idea worked like a charm.
The actual deployment site is now a fully operational demo, in the sense that there are no major features left to figure out an implementation for. There’s a backend that collects, stores, and cleans data. There’s a front end that renders charts and tables and runs smoothly enough. And its all running on my Linode server, there for the .htpasswd
-bearing world to see. Huzzah!