Rate limiting in Node.js with Promises

I’ve been re-familiarising myself with Node.js recently, and there’s a lot to catch up on since I last used it in anger. With the support for ES6 I’ve been looking through the changes listed and playing with features here and there to get acquainted with what can now be done. While many seem very nice when used in toy projects, I haven’t yet used them for a large enough system to say which ones may end up more trouble than they’re worth. Enter Promises.

I won’t go into the mechanics of Promises here, as there are already a good number of articles and references out there already, but instead I’ll look at a particular use case that I had. I was working on a small script to call a rate-limited REST API and thought this might be a good time to try a Promise-based library rather than the callbacks I’d used in the past. Fortunately rest exists, straightforward to use and with decent documentation to boot. Calling the API is as easy as:

Once the libraries are imported, the rest client is wrapped in Interceptors that each change how the call will be made, first adding a MIME type, prefix for the path, then an extra header. Finally we return client(options), which initiates the call and returns a Promise.

This was working pretty well by itself, but I still needed to introduce some rate limiting. To simplify my experiments I thought I’d create a very basic Promise that did nothing but log to the console, and then introduce the rest call later, so I set that rest call aside and started out with a very basic Promise:

As an initial limit, I thought one call every two seconds would be fine for testing. Wrapping setTimeout in a Promise, I can create a delay function:

To get the call happening at most once during each interval was fairly straightforward. Looking at Promise.all, it resolves only once all its promises have resolved or is rejected as soon as any one of those is rejected. This meant that I could call it with my task and the delay:

So now if I could just figure out how to repeatedly call it I could have the task running at most every two seconds, or longer if the task itself exceeds that time frame. I did find some suggestions on how to do this, and implemented a small loop using recursive calls:

This worked quite well, logging and then delaying for two seconds, then calling loop() again once Promise.all had resolved. Great! But this still bugged me a little, and there was a niggling doubt in the back of my mind. I wanted this to keep looping, after all, and I wasn’t confident enough in my understanding of the mechanics of promises to be sure that it wasn’t going to blow the stack, or create some issue with memory leaks with infinite Promises. Each loop created a new Promise that was resolved through calling and returning loop again. Doing some more digging, it did appear that I still might have a problem. This was confirmed when I found an interesting issue in the node.js issues list.

Reading through the issue, it did seem that using this pattern with Node.js native Promises would indeed cause a memory leak, as it continues to create a resolve chain of Promises until the loop has completed, which in this case would be never (or when the program stops). For a loop with a definite time frame it’s a non-issue as the memory is freed once resolution occurs, but for indefinite or long running loops it’s potentially problematic. As discussed in the github issue you can get around this by removing the return, which means that the loop continues but the Promises aren’t chained through resolution.

If you want to use any result as a Promise that solution obviously won’t work, but I found that bluebird‘s implementation of Promises did not have this behaviour and could be run indefinitely without increasing memory. It is able to do this by using an implemenation that varies slightly from the Promises/A+ spec that Node.js adheres to but not in a way that affects the semantics of the operations in which I was interested. It also provides a number of handy features for simplifying Promise-based workflows, including promisifying callback-based functions and composition of Promises. I didn’t delve too much into those for this exercise though, but at least I did finish my working loop:

You can also look at this bluebird issue to see some more examples. To get an idea of how the Node.js and bluebird promises compare, you can remove/reduce the delay, make the counter incremental and then add:

The Promise const can be commented out to use the Node.js Promise, or left in to use bluebird’s.

Posted in Programming | Tagged , , , | Leave a comment

Remote Profiling the JVM through an SSH Tunnel

This one took a little time to figure out. I wanted to profile a remote JVM application using VisualVM locally, in a way that didn’t expose any ports to the outside world and has a reasonable level of security. An SSH tunnel to access it would be fine, but I had to figure out whether the tools supported it.

VisualVM is a tool I’ve used at various points in time to find some of the more obvious performance issues in my software. While not as fully featured as other tools out there, it can still be useful when looking for some of the more egregious bottlenecks. Using VisualVM to profile a local JVM application is straightforward enough – start up the application, and it appears in the Local list under the Applications tab. Profiling a remote application is where it gets a little more tricky.

Oracle’s JVM has a range of instrumentation available for remote monitoring, and one option is through the JMX agent. Since Java 6, any application run on the JVM should have the Attach API, which means that to open remote monitoring on a given port you just need to set the appropriate system property on startup, ie:

By default, the protocol connecting a JMX client to a JMX server is Java RMI. Now it used to be that the registry ran on the specified port and a separate, roaming data port was created. You can probably find several rants discussions on older blogs around about various ways to find or pin this down, but since Java 7u4 you can specify the RMI port, even binding it to the same port as the registry, with a system property:

The JMX/RMI server now listens on the one port, though it’s bound to 0.0.0.0 by default and I found no way to specify a property to force it to localhost. While this *should* be fine in a local network or a firewalled server, I do prefer to bind to localhost when I can. To do this you can specify your own agent that sets up the server programmatically, and for this I used https://github.com/gehel/jmx-rmi-agent. This agent runs on startup and creates the JMX service with an explicit URL, and when you set the option to force it to bind to localhost it uses “service:jmx:rmi://localhost:3000/jndi/rmi://localhost:3000/jmxrmi”. You use the -javaagent option to set it as the agent, and it has its own set of system variables so as not to clash with the defaults, but they are fairly straightforward:

Unfortunately it was still externally visible even with the explicit localhost URL, but I did some more reading and found that by creating a custom RMIServerSocketFactory and passing that to the LocateRegistry.createRegistry call, you can bind the socket yourself. Because that’s what we really enjoy having to do. At any rate, my changes are available on the custom-socketfactory branch of my fork, at https://github.com/taufiqkh/jmx-rmi-agent/tree/custom-socketfactory.

All this done, it was binding to 127.0.0.1 like a champ:

Now, some authentication would be nice. It’s bound to localhost and only authorised users should have access to the system, but I do like layers in my security. Fortunately this part was pretty simple. I was happy to use the default monitorRole security role and to specify a password all I needed to do is create a jmx password file with the following contents:

The password file is specified with the system property:

The final command line was the combination of all these options:

System up and running, I now needed to configure the tunnel – this part was fairly straightforward. I was using putty, which has various options for tunnelling. All I needed to do was to set the port it would listen to on my desktop and the port to which it would forward on the remote system, and press Add:

Putty Tunnel

Going back to VisualVM, I now needed to set up the connection. I selected File->Add JMX Connection… and added the URL output by the jmx-rmi-agent, service:jmx:rmi://localhost:3000/jndi/rmi://localhost:3000/jmxrmi. Once I filled in the username and password and selected Do not require SSL for this connection (that’s a bit of configuration I didn’t want to get into at this stage) I pressed OK:

visualvm_add_jmx_connection

This done, I selected the remote system in the Applications tab, and up came the overview. I moved to the Monitor tab for a look at the current activity:

visualvm_remote_connection

Yak shaving done, it’s time for the real work…

Posted in Programming | Tagged , | Leave a comment

Could not find or load main class

I recently came across what seemed to be a fairly common error message while trying to run a Java jar on a newly installed Ubuntu 16.04 VM:

This is a pretty common issue that usually stems from incorrect classpaths or a wrongly specified main class, which is what appears when you search for “Could not find or load main class”. I’d checked the usual suspects though, checking for:

  • typos in the file name – java doesn’t tell you if any of the classpath entries don’t exist
  • existence of the class – I unzipped the file to check that yes, the class exists
  • typos in the package or class -copied the package and class name from the source just in case

All these checked out, so what was the issue? I noticed that there was no META-INF/Manifest.MF, which would causes java -jar to fail but doesn’t cause any issues when you’re explicitly specifying a main class – or does it? I read through the jar file specification to see if perhaps the jar wasn’t properly packaged but it indicated that META-INF and its contents were optional in this situation. Hmmm…

Double checking the WurmServerGuiMain class, I verified again that it had the correct main method signature but this time I remembered that it was a Java FX class that subclassed the javafx.application.Application class. Java FX has caused me issues in the past, partly due to the way it was introduced as an add-on ext lib, and partly due to the way it gets loaded in the case of a jnlp, but this was just a bog-standard jar file so…

Wait, do I need to explicitly include jfxrt.jar? If the parent Application class couldn’t be found, that’d be another reason for being unable to load the main class. Looking in $JAVA_HOME/jre/lib/ext, I couldn’t see the jar, so I did a little reading and found out that on Ubuntu 16.04 at least, OpenJFX is in a separate package and is not included with the OpenJDK 8 JRE. Now we’re getting somewhere! Now a quick

and I’m able to run the jar. Still more bugs, but at least those ones are expected.

So, two points that I’ve learnt:

  • “Error: Could not find or load main class” is a more non-specific error than I thought, and can mean that your parent class cannot be found (though it won’t tell you that explicitly)
  • Java FX does not always come included in the OpenJDK 8 JRE

Both seem fairly straightforward in hindsight, but at least writing it out should help in troubleshooting similar situations for myself and customers in the future.

Posted in Programming | Tagged , | Leave a comment

Setting up Rocket.Chat in userspace on Ubuntu 12.04

Not too long ago I needed to set up Rocket.Chat for internal evaluation. At the time its installation process was not especially well documented and required some existing knowledge of its underlying stack in order to figure out how to get it running on some of the less supported systems, so there were a few blocks that needed sorting out. After I put together a set of steps for internal use, someone suggested that it might be useful to post externally, so here are the steps I took to get it running as non-root on a Ubuntu 12.04 VM.

Note: At the time of writing Rocket.Chat was in a fairly early stage; these steps will most likely be replaced by far more straightforward installs as it matures and it has already expanded its set of supported environments considerably.

Node install

Rocket.Chat runs on node and I was a little unsure of which particular version I needed. I also wanted to see if it could run without needing root, so I set up nvm first to allow easy switching of node versions and the ability run it as a user account.

First, download and install nvm:

Relog so that the changes take effect.

As I found out later, the version of Meteor used by Rocket.Chat didn’t support later versions, so using nvm I installed node.js 0.10.42:

MongoDB install

For its persistence Rocket.Chat uses MongoDB. Again, to run in user space we need to download and prepare the environment ourselves. Note that this setup runs MongoDB insecurely and these steps need to be modified for an externally visible system.

To later shut down mongodb:

Finally, Rocket.Chat install

Now, install Rocket.Chat:

Set up its environment variables. These will also need to be added to your .bashrc or similar profile in order to have them next time you login:

Now we should be good to go

Initially I tried to install later versions (v4 and v5) of Node.js before I realised that they weren’t supported. I had some issues that came up, so while the current version of Rocket.Chat doesn’t use these versions it’s worth noting that on Ubuntu 12.04 you need to update gcc and the g++ compilers (v5) (see here for an explanation). This is not necessary for later versions of Ubuntu that should already have them.

 

Posted in ops, Programming | Tagged | Leave a comment

Installing a Let’s Encrypt certificate for ZNC

I’ve been running ZNC, an IRC bouncer that I use for maintaining a presence in channels while I’m offline and keeping logs so I don’t miss conversations. As it’s been using a self-signed certificate for a while, I thought I’d finally get around to creating a proper domain validated certificate using Let’s Encrypt.

As the system on which I was running ZNC was too old to run the official Let’s Encrypt client, I opted for a shell-script-only version. First, install acme.sh (and it’s generally good practice to audit it so that you know what it’s doing):

Now that’s done, create the account and domain keys:

Domain certificates? Account keys? What’s all this, then?

Let’s Encrypt “is a free, automated, and open certificate authority brought to you by the Internet Security Research Group (ISRG)”. The service and its clients provide an implementation of the ACME protocol (draft here), which describes an automated process for the verification and the issuance of domain certificates. The certificates issued are Domain Validated, so they can only say that you are in control of the domain, and can’t give you any more information such as the legal entity responsible for the domain, or any verification that the requester of the certificate was affiliated or representative of that legal entity. So basically, you control the domain, but no information on “you”.

Your account key, created with the –createAccountKey option, generates a private key that will be used to register an account and sign subsequent requests under that account. Using –createDomainKey generates a private key that is used to sign specific domain certificate requests. Running acme.sh –createCSR creates a certificate request, which is a request for a certificate that is signed by both the account private key and the domain private key. So far this has all been local preparatory stuff, with no communications with the issuing server.

Now that the preparations are complete, it’s time to contact the Let’s Encrypt service, register our account and make the request. When the request is made, the issuing service responds with a challenge to prove that you are in control of the domain, by requiring you to sign some challenge tokens and provide them as files over HTTP/HTTPS at the .well-known/acme-challenge/ url path of your domain. Run the issuing script with a standalone server to request the certificate and provide those challenge responses. Sudo is needed as it listens on port 80:

As you’ll note in the output, this will store your certificate in ~/.acme.sh/domain.example.com. Concatenate the certificate and domain key into a znc.pem file and copy to the .znc directory:

Visit your znc admin page in your browser and you can see it in action.

A good demonstration and explanation of what is happening, step by step, can be shown in the README.md of Daniel Roesler’s letsencrypt-nosudo. This script does the work of determining what steps need to be taken and displays the command commands that the user can manually execute in order to perform each step.

Edit: Added explanation of what’s happening under the hood.

Posted in ops, Software | Tagged , | Leave a comment

Auto-syncing Wurm map dump to web server

In my downtime I help host KangaWU, an Australia-based Wurm Unlimited server. One of the things we needed was auto-generated map dumps syncing to our web server to allow frequent updates of the online map. Looking at the options available, I decided on using rsync in a cron job, with public/private key pairs for authentication.

The first roadblock was that since cron runs a limited environment, it doesn’t have access to any passphrase entered into ssh-agent. Using keychain installed in user-space works so that the passphrase can be entered in a terminal session and still be accessible to cron jobs. So:

  • Since keychain isn’t in a CentOS 7 package, downloaded the tarball and extracted to $HOME/util/keychain-2.8.2
  • Created a refresh-keychain script to start keychain if it hasn’t already been started:
  • Made sure the above script was in PATH and executed in .bashrc
  • Executed it for the first time, to enter the passphrase

This does mean that on server restart the passphrase needs to be entered, hence adding it to .bashrc so that when that happens you get asked. Now that’s done, I thought I’d add a .ssh config file to make sure that I don’t need to specify an identity file each time I connect:

  • Create a new file $HOME/.ssh/config, with the following:
  • Verify that it works, with ssh yourwebhost.com

Write up a short script to rsync the files. Note the trailing / at the end of the source, which indicates that the contents of that directory are to be copied to the destination directory, rather than creating the source directory within the destination:

Now, a script for cron to call:

Add a cron entry to run the script at 4:30am each day and we’re done:

While at some point a live map would be ideal, this does the job for now, with maps being at most 24hrs out of date.

Posted in Software | Leave a comment

Porting the Netty DiscardServer example to clojure

I recently looked at the Netty library as the basis of a server for work, so after a recent LambdaJam I was inspired to write their DiscardServer example in Clojure, on a basic, line-to-line basis. Has some issues, not the least of which is that at the time of writing their example is missing sections, and doesn’t compile. Was able to guess my way around enough of it to get it working in Java first:

Initial attempts writing the Clojure had some problems. There was some confusion due to a bug in clojure which didn’t correctly call the public, final method of a non-public, abstract superclass. This meant that the call to ServerBootstrap.channel() failed, an issue that I worked around by writing a static Java method to call the original method:

Forgot about Clojure/java varargs calls, and the way that Java’s variable argument methods actually used an array under the hood. This confused me for a while, as my initial attempt to call ChannelPipeline.addLast(ChannelHandler…handlers) resulted in ClassCastException:

One workaround is to create the array with a single value:

but this seemed fairly clumsy, exposing some messy implementation details to get things working. Still not a great solution, I instead decided to write another Java method to call the original:

This make the Clojure call a little clearer, though I can go back to the original array creation if I do need more than one handler:

End result, the Netty java class for working around interop issues:

and the Clojure:

Posted in Programming | Tagged , , | Leave a comment

Creating numeric types from byte arrays in Clojure

While writing my Clojure-based Minecraft map reader and NBT library, carrit, I’ve need to do some messing around with binary files. For this library I’ve made the decision to implement what I can in Clojure’s available functions, dropping down into Java interop where I haven’t found the native Clojure equivalent. As a minecraft map is divided into region files that are each relatively small (~3MB for larger ones) I’ve been slurping them one by one into a single contiguous byte array, decompressing each individual Minecraft sector of that into separate byte arrays that are then read and interpreted by my NBT code.

Been having some issues with Clojure’s interpretation of bytes when trying to convert a section of a byte array into a numeric value. For example, given an array [0xFF 0xFF 0xFF 0xFF]:

Results in:

As expected. However, given an array [0x00 0xFF 0xFF 0xFF], which should result in 16777215, we instead get:

Fortunately, I’m not the only one who’s encountered this, as can be seen in the following threads:

Some things to note are that bit-left-shift performs a widening conversion on the byte to a long and preserves the sign before shifting left, and that bit-or performs a similar widening conversion. So what was happening in the above example was:

  1. Retrieval of the first byte, 0x00. This is interpreted as 0. bit-shift-left shifts automatically performs a widening conversion, preserving the sign, and shifts this left by 24 bits, resulting in 0 as expected.
  2. Retrieval of the second byte, 0xFF. This is interpreted as -1. bit-shift-left automatically performs a widening conversion, preserving the sign, and shifts this left by 16 bits, resulting in a long with a value of -65536.
  3. Retrieval of the third byte, 0xFF. This is interpreted as -1. bit-shift-left automatically performs a widening conversion, preserving the sign, and shifts this left by 8 bits, resulting in a long with a value of -256.
  4. Retrieval of the fourth byte, 0xFF. This is interpreted as -1.
  5. Before the bit-or, 0xFF is interpreted as -1 and widened.
  6. (bit-or a b c d) performs OR on a and b, and again between the result and the next argument, repeating this for successive arguments. In this case, it perfoms bit-or on 0, -65536, -256 and -1, which of course results in -1.

Certainly fun when you don’t know what’s going on. One way around this is to manually perform the conversions yourself, then bit-and with 0xFF to ensure that the sign is removed, ie.:

A little convoluted and can do with some tidying up, but seems to do the job for now.

Edit: Fixed formatting of results, clojure brush doesn’t like raw numbers.

Posted in Programming, Software | Tagged , , , | Leave a comment

Migrating from duck-streams/to-byte-array in Clojure 1.3

Just spent some time migrating my minecraft NBT library from to Clojure 1.2.1 to 1.3, had to make a few changes as many clojure.contrib 1.2.0 libraries were moved to separate modules or merged into core. The reliance on clojure.contrib.duck-streams was a tricky one to get around. Most of its functionality was moved to clojure.java.io, but not to-byte-array which was previously used to slurp in the binary .mca files. No much documentation around, but with a few hints from Stack Overflow I ended up with the following:

Seems to work so far, passing the current regression tests.

Posted in Programming, Software | Tagged , | Leave a comment

HGFS on a Ubuntu 11.04 VM

Had some issues sharing a windows host directory with my Ubuntu 11.04 VM on VMware 6.5.5 – kept coming back with a “Unable to update run-time folder sharing status: Unknown Error”, which was unhelpful to say the least.

I’d previously followed the instructions on installing VMware Tools so I expected it to just work. After reinstalling and still getting the errors, I found some additional instructions on building and installing the hgfs module. After apt-get installing open-vm-source and running module-assistant to install, rebooted the machine. Enabling folder sharing from the host didn’t seem to give any errors, and the folder was able to be loaded.

Posted in Programming | Leave a comment