Earlier this week I was working with Domino REST API for a personal project and encountered what appeared to be a bug. It was a very strange issue, but one that had a simple cause that was ultimately easy to verify. Shortly after joining HCL I wrote a blog post on troubleshooting support. If you didn’t read that blog post at the time or need a refresher, it’s probably one of the most important blog posts I’ve ever written and most of what I covered in that blog post was relevant to solving this problem. Coincidentally, what I wrote about understanding in my most recent blog post was also crucial.
The “Bug”
First off, I regularly see developers hit a problem and instantly claim it’s a bug, often with little investigation and even less understanding of the process. Worst still, developers are typically at the receiving end of such claims. You would think they would be more judicious or maybe it’s a touch of hubris.
Unless I have looked in depth at the code - and even often when I have - I would hope I always say that I think I’ve found a bug. After all, the way I think I expect the code to work may not be what was intended and rarely is a system commented with total clarity. Key to proving a bug is reproducing the symptoms, and key to reproducing the symptoms is minimising external factors and creating a simple reproducer.
In this case, Domino REST API’s `/lists/{name}’ endpoint was resulting in unexpected results. For certain views on the server I was using, e.g. a view with a name “vwFoo” was returning expected results. But for hidden views, e.g. “(luKeys)” I was getting a 405 “Method Not ALlowed” error when accessing via Postman.
Verify Behaviour From Another Entrypoint
Domino REST API provides Swagger specs (/openapi/index.html). This allows you to test authentication and test APIs, generically or for a specific endpoint. This was a part I coded many years ago, to allow developers to share the OpenAPI specification with third parties and also test live on the server. I’ve used DRAPI from Postman for many years, I helped review the Postman collections and environments before the early access release. So I was fairly certain I had not made a mistake, I thought the brackets for the hidden view name needed escaping and I thought I had done it correctly. But the Swagger page allows me to test programmatically without the chance of user error, by selecting the view name from a combo box instead of typing the name manually. It also lets me see the curl command used, which I can then use elsewhere and allows me to cross-reference the intended URL. This confirmed the URL format - /lists/%28luKeys%29?dataSource=myScopeName.
Postman has an option to view the actual requests and responses, by going to the menu and selecting View > Postman Console. This confirmed what was being sent from Postman was the same as the website, and both approaches returned 405 Method No Allowed error.
Know Expected Behaviour
There are a few known returns, which we can easily verify:
- if it’s right, content is returned.
- if there are no documents accessible, an empty array is returned.
- if the scope name is wrong, an error “A scope definition with that name cannot be found” and 404 response code is encountered.
- if the view name does not match something configured, the error says “The list is not configured for this database” and a 404 response code is encountered.
- using a verb other than GET gives a 405 Method Not Allowed error.
But the viewI was using was definitely GET.
Verify Behaviour in Another Environment
If you do not already use Domino on Docker, I would highly recommend it for quickly setting up an environment with standard setup. It can help reproduce in a vanilla environment. Checking a hidden view in a different database on a different server did not reproduce the problem. But the server was slightly more up-to-date. The difference might be important. Upgrading the server to the latest official release took a little longer for various PEBKAC reasons and trying to set up a debuggable server to step through the code - a luxury not all have - hit some different challenges with pro code developer environment setup. Fortunately someone else was able to confirm they could not reproduce the problem.
Understand the Process
Here we encounter the most fundamental rookie mistake I made. I saw DRAPI in my browser, saw Postman request and response, knew DRAPI was on the server and assumed no other relevant steps. Rookie error, there’s always a process. This particular server had nginx as the proxy server, a fact I remembered when prompted by my colleague.
Know Your Logs
Looking at the access logs for nginx showed the 405 error being returned there and the URL that was being sent, with %28 and %29.
Next step was to look to see if the request was being blocked by nginx or routed to DRAPI. The point of reference was the domino-keep.log in IBM_TECHNICAL_SUPPORT directory. Here I saw logging that verified the request was hitting DRAPI, had a url /lists/(luKeys)?dataSource=myScopeName and, more importantly, the 405 response was being sent from DRAPI.
Know Expected Behaviour Part Two
Again, at this point, the assumption may be a DRAPI bug. But I had not verified expected behaviour. There are two ways to do so:
- Test in a working environment and compare logging.
- Test without intermediaries.
My colleague, Ron, was able to share his log output and proved that DRAPI expects the view name without URL decoding, so “%28luKeys%29”, not “(luKeys)”. Running the curl command (remember what the Swagger page provided us?) from the server Domino was running on, using localhost instead of the host name, allowed me to confirm the logging. It also confirmed the problem was not a bug, and not DRAPI, but nginx configuration.
The Cause
So nginx was decoding the %28 and %29 rather than just passing the URL as received. I’m not an nginx expert, so some googling led to a variety of StackOverflow questions. At some point I hit this answer or one similar. The configuration had:
location /api {
proxy_pass http://localhost:8080/api;
}
The solution was changing it to:
location /api {
proxy_pass http://localhost:8080;
}
This prevented nginx decoding (or normalising) the path parameter, which avoided the 405 error. Problem solved, no bug, environmental issue, not caused by DRAPI, just triggered by a call to DRAPI and returned by DRAPI.