Sergey Maskalik

Sergey Maskalik's blog

In the pursuit of mastery

I’ve recently updated a tiny utility script (sqsmover) that I wrote when I was learning Go Lang that moves AWS SQS messages between queues. After three years, it is still a userful tool when I need to move messages between queues. I feel like it could be useful to others who use SQS and deadletter queues because at one point something is going to fail and your messages will end up in the deadletter.

In order to polish it, I’ve included the following features:

Help menu

sqs mover help menu

Region flag

Original version had region hardcoded in the code and needed a code change if you used a different region, oops. I’ve modified to allow region to be overwritten as a command line flag and default to us-west-2.

Installation

Updated installation instruction with one line shell script install for macOS, Linux, or OpenBSD! And compiled binary for Windows. No longer in order to install the app you have to have Go Lang installed locally and issue a go get ... command. This was pretty easy to do with awesome goreleaser project and companion godownloader. GoReleaser automatically creates binary releases and publishes it to github while GoDownloader will create a bash script for one line installation.

Intuitive error messaging

It sucked when your script fails and you don’t know what went wrong, so I’ve included a lot more user friendly error messages that should help users with figuring out what needs to be done.

error

Progress indicator

For the first version app I used to just output every message moved to the screen. This is not very useful if you have thousands of messages. A better user experience is to include a progress indicator.

progress indicator

Future work

At the moment the utility is using a batch download and upload which moves about 10 messages at a time. It’s pretty fast if you need to move few thousand of messages but if you have a million stuck in the deadletter queue I could imagine it could take a while. To speed up, I’m thinking to introduce multiple receiver and consumer threads and allow users to specify how many threads to use.

That’s it let me know if you come accross any issues using github, thanks!

When used to best advantage, exceptions can improve a program’s readability, reliability, and maintainability. When used improperly, they can have the opposite effect. - Joshua Block

Most of this wisdom comes from Joshua Bloch’s Effective Java book, see the book for the complete discussion on the exceptions.

Use exceptions only for exceptions conditions

// Horrible abuse of exceptions. Don't ever do this!
try {
  int i = 0;
  while(true)
    range[i++].climb();
} catch (ArrayIndexOutOfBoundsException e) {
	...
}

Better idiom for looping through array is: for (Mountain m : range) m.climb();

  • Because exceptions are designed for exceptional circumstances, there is little incentive for JVM implementors to make them fast as explicit tests.
  • Placing code inside a try-catch block inhibits certain optimizations that modern JVM implementation might otherwise perform.
  • Exception based idiom is much slower than state checking.

A well-designed API must not force its clients to use exceptions for ordinary control flow

Throw exceptions appropriate to the abstraction

Problems when a method throwing an exception that has not apparent connection to the task it performs, usually propagates by lower level abstraction.

  • Pollutes API of the higher level with implementation details
  • If implementation of the higher layer changes, the exceptions that it throws will change too, potentially breaking clients.

Higher layers should catch lower-level exceptions and, in their place, throw exceptions that can be explained in terms of the higher-level abstraction. This idiom is know as exception translation.

// Exception translation
try {
   httpClient.execute(...)
} catch (IOException e) {
   throws PspApiException(...);
}
// Exception chaining (special form of translation)
try {
   httpClient.execute(...)
} catch (IOException cause) {
   throws PspApiException(cause);
}

class PspApiException extends Exception {
   PspApiException(Throwable cause) {
      super(cause);
   }
}

While exception translation is superior to mindless propagation of exceptions from lower layers, it should not be overused.

  • Use validate params
  • Deal with exceptions on lower level

Use checked exceptions for recoverable conditions and runtime exceptions for programming errors.

The cardinal rule in deciding whether to use a check or an unchecked exception is this: use checked exceptions for conditions from which the caller can reasonably be expected to recover.

Use runtime exceptions to indicate programming errors. Great majority is precondition violation, a failure of the client to adhere to the contract established by the API specification.

Because checked exceptions generally indicate recoverable conditions, it’s especially important for such exceptions to provide methods that furnish information that could help the caller to recover. For example, redemption of the card has failed because there is no money left, that information could be relayed back to player.

More on where to use checked vs unchecked.

Avoid unnecessary use of checked exceptions

Overuse of checked exceptions can make an API far less pleasant to use. Forces caller to handle the exception.

Burden is justified if the exceptional condition cannot be prevented by the proper use of the API and the programmer using the API can take some useful action once confronted with the exception. Unless both of these conditions hold, an unchecked exception is more appropriate.

Favor the use of standard exceptions

IllegalArgumentException (Non-null parameter value is inappropriate) IllegalStateException (Object state is inappropriate for method invocation) NullPointerException (Parameter value is null where prohibited) IndexOutOfBoundsException, ConcurrentModificationException, UnsupportedOperationException

Document all exceptions thrown by each method

To use a method properly you need to know about exceptions that it throws, therefore, it is critically important that you take the time to carefully document all of the exceptions thrown by each method.

Always declare checked exceptions individually, and document precisely the conditions under which each one is thrown using the Javadoc @throws tag

Unchecked exceptions represent programming errors so it is wise to document them so programmers get familiar with all the errors so they can avoid it.

If an exception is thrown by many methods in a class for the same reason, it is acceptable to document the exception in the class’s documentation comment.

Include failure-capture information in detail messages

To capture the failure, the detail message of an exception should contain values of all parameter and fields that “contributed” to the exception.

public IndexOutOfBoundsException(int lowerBound, int upperBound, int index) {
  // Generate a detailed message that captures the failure
  super("Lower bound: " + lowerBound + ", Upper bound: " + upperBound + ", Index: " + index);

  // Save failure information for programmatic access
  this.lowerBound = lowerBound;
  this.upperBound = upperBound;
  this.index = index;
}

Don’t ignore Exceptions

An empty catch block defeats the purpose of exception.

Exceptions code smells

try {
   // 100 lines of code
  ...
} catch (Exception e) {
  //Something failed.. not sure what or where, is it important or is it a bug, who knows ¯\_(ツ)_/¯
}

Ideally try catch around the line where exception is expected or a few lines to make the code more readable.

Catch all unhandled exceptions in the global error handler and log

Unhandled exceptions indicate programming errors, most likely the API was not used correctly and that’s a bug that could be fixed

I found it straight forward to configure expressjs to send http requests to logs and to setup log parsing. If you use log4js, there is a connectlogger that can capture expressjs/connect http requests. Finally, to make sense of logs I prefer to use ELK Stack with Logstash parsing the incoming logs, Elastic Search indexing, and Kibana for functional dashboards.

I always like to include the response times of http requests. It helps with troubleshooting performance issues down the line. Below is the slightly modified default format with response time appended to the end.

var httpLogFormat = ':remote-addr - - [:date] ":method :url ' +
					'HTTP/:http-version" :status :res[content-length] ' +
                    '":referrer" ":user-agent" :response-time';
app.use(log4js.connectLogger(log4js.getLogger("http"), { level: 'auto', format: httpLogFormat }));

Example log output:

10.201.44.200 - - [Tue, 29 Sep 2015 04:52:53 GMT] "GET / HTTP/1.1" 302 66 "" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36" 2

Logstash Log4js appender

Using log4js-logstash appender, I specify the following in the log4js config.

{
  "appenders": [
	{
	  "type": "log4js-logstash",
	  "host": "10.0.0.10",
	  "port": 5959,
	  "fields": {
		"app": "my-app",
		"env": "dev"
	  }
	}
  ],
  "replaceConsole": true
}

Fields are nice to have if you want to tag your logs with application name or environment, so you can tell where the logs are coming from.

Logstash parsing

In order to parse our custom log we need create a logstash pattern, and place it into the /etc/logstash/patterns folder.

Here is the pattern for parsing the log format above

EXPRESSHTTP %{IP:clientip} - - \[%{DATA:timestamp}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|undefined) \"%{URI:referrer}?\" \"%{DATA:agent}\" %{NUMBER:response_time}

Logstash input and filter

input {
  tcp {
	codec => "json"
	port => 5959
	type => "my-app"
  }
}
filter {
  if [type] == "my-app" {
	mutate {
	  rename => { "[@fields][app]" => "app" }
	  rename => { "[@fields][category]" => "category" }
	  rename => { "[@fields][env]" => "env" }
	  rename => { "[@fields][level]" => "priority" }
	  rename => { "category" => "logger_name" }
	  rename => { "@message" => "message" }
	}
  }
  if [type] == "my-app" and [logger_name] == "http" {
	grok {
	  patterns_dir => "/etc/logstash/patterns"
	  match => { "message" => "%{EXPRESSHTTP}" }
	  add_field => ["received_at", "%{@timestamp}"]
	  remove_field => ["message"]
	}
	date {
	  match => ["timestamp", "EEE, dd MMM YYYY HH:mm:ss zzz"]
	  remove_field => "timestamp"
	}
  }
} In the above config, I perform a number of renames on the nested fields so it can have short and clear names.  And in the final if clause, I checks if the logger_name is marked with "http" and then perform a grok parsing of the logs. It's also good to add received time and then parse the actual log time as in the above date filter.