Sergey Maskalik's Blog
Sergey Maskalik

Sergey Maskalik's blog

In the pursuit of mastery

I found it straight forward to configure expressjs to send http requests to logs and to setup log parsing. If you use log4js, there is a connectlogger that can capture expressjs/connect http requests. Finally, to make sense of logs I prefer to use ELK Stack with Logstash parsing the incoming logs, Elastic Search indexing, and Kibana for functional dashboards.

I always like to include the response times of http requests. It helps with troubleshooting performance issues down the line. Below is the slightly modified default format with response time appended to the end.

var httpLogFormat = ':remote-addr - - [:date] ":method :url ' +
					'HTTP/:http-version" :status :res[content-length] ' +
                    '":referrer" ":user-agent" :response-time';
app.use(log4js.connectLogger(log4js.getLogger("http"), { level: 'auto', format: httpLogFormat }));

Example log output: - - [Tue, 29 Sep 2015 04:52:53 GMT] "GET / HTTP/1.1" 302 66 "" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36" 2

Logstash Log4js appender

Using log4js-logstash appender, I specify the following in the log4js config.

  "appenders": [
	  "type": "log4js-logstash",
	  "host": "",
	  "port": 5959,
	  "fields": {
		"app": "my-app",
		"env": "dev"
  "replaceConsole": true

Fields are nice to have if you want to tag your logs with application name or environment, so you can tell where the logs are coming from.

Logstash parsing

In order to parse our custom log we need create a logstash pattern, and place it into the /etc/logstash/patterns folder.

Here is the pattern for parsing the log format above

EXPRESSHTTP %{IP:clientip} - - \[%{DATA:timestamp}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|undefined) \"%{URI:referrer}?\" \"%{DATA:agent}\" %{NUMBER:response_time}

Logstash input and filter

input {
  tcp {
	codec => "json"
	port => 5959
	type => "my-app"
filter {
  if [type] == "my-app" {
	mutate {
	  rename => { "[@fields][app]" => "app" }
	  rename => { "[@fields][category]" => "category" }
	  rename => { "[@fields][env]" => "env" }
	  rename => { "[@fields][level]" => "priority" }
	  rename => { "category" => "logger_name" }
	  rename => { "@message" => "message" }
  if [type] == "my-app" and [logger_name] == "http" {
	grok {
	  patterns_dir => "/etc/logstash/patterns"
	  match => { "message" => "%{EXPRESSHTTP}" }
	  add_field => ["received_at", "%{@timestamp}"]
	  remove_field => ["message"]
	date {
	  match => ["timestamp", "EEE, dd MMM YYYY HH:mm:ss zzz"]
	  remove_field => "timestamp"
} In the above config, I perform a number of renames on the nested fields so it can have short and clear names.  And in the final if clause, I checks if the logger_name is marked with "http" and then perform a grok parsing of the logs. It's also good to add received time and then parse the actual log time as in the above date filter.

I’m a big believer in continuous improvement and personal growth. But how do you know if you are actually making progress in your life and becoming better? Are you moving towards your goals or are you full of self-delusion that keeps you at bay?

I can tell you from personal experience, that we can be dishonest with ourselves. It may seem like you are actually moving forward by reading books, talking to your friends about your goals, and making up strange tasks that you need to do before you actually do what you need to do. You might tell yourself, I “If I knew this or that, or had this amazing experience, then I would be ready and able to do it. But there is never going to be a perfect moment! It’s always going to be a moving target and we need to start acting toward, and working on, our goals. Bottom line – we need more doing, something that will actually move us towards the goal.

We need to make mistakes in order to learn. But we cannot fail by being safe and not putting ourselves out there. Hence we grow by doing and learning from being foolish.

At this moment I feel like I need to make mistakes and learn from them. I want to challenge myself and accomplish my goals. That is why I’ve decided to try personal coaching. Here are a few things that a good coach should be able to help you with:

Outside Opinion

Your coach is not your friend, she doesn’t know who you are and she is not going to accept your excuses or be nice to you. She will only look at the results that are being produced rather than excuses, and keep you accountable. In addition, she can provide an unbiased opinion about how honest you are being with yourself.

Keep you accountable

It’s one thing to make a promise to yourself that you are not going to keep, but making a commitment about your goals to another person who you respect, will make you work harder.

Your best interests

Your coach’s motivation is to help you succeed with your goals. Her interests are aligned with your best interests. She is there to support you and move you along the path to your goals. We all want the best for ourselves, but we are also very easy on ourselves at times, which does not produce results.

Work out psychological issues

A lot of times the problems that we experience are only in our heads, and fear of failure stops us before we can continue to grow. An experienced coach should be very good at helping you to work out those issues.

Deliberate practice

Just like in sports, a coach will make you do drills that will help you when it’s show time. In addition, she will help you measure your performance.

Getting clear on your goals

Starting from the end result, a coach should be able to help you work out what the most important thing in your life is. Are you chasing someone else’s dreams, or is this something you truly want for yourself?

Enjoy your life now

Goals are great, but if you are not enjoying what you are doing, what is the point of living? A coach can, at times, also remind you to have fun and live in the present if she notices that you are getting burned out.

I’m looking forward to working with a coach and putting myself in a higher gear on the road of greater fulfillment and personal growth.

One of the first questions non-technical entrepreneurs who have an idea ask me is, “How much would it cost me to build an application and how soon can you build it?” And for someone, who is not familiar with software development, it could be very difficult and even frustrating to get a realistic estimate with a timeframe. You might be surprised to learn that a lot of developers and managers don’t understand how to estimate software either! I’ve personally been a part of large scale development efforts, with ballpark estimates, coming from management, where projects never met the deadlines and were delivered way over budget. Hopefully after reading this post you will have a better understanding of what it takes to get a good estimate.

Software Estimation Story

Building software is much like building a house. Imagine you want to build a house and have a friend who is an architect, so you ask him if it’s possible to build a three bedroom house for around $100,000. He thinks about it and says yes, but it depends on the type of house and what’s going to be in it. In your friend’s mind he imagines a simple house, with no thrills, that he could design and build for that amount. But you might have some other ideas. You want to have a nice swimming pool, a playground, a three car garage, bamboo floors, granite everywhere, a sauna, gold plated mirrors, high ceilings, and etc. You also might want a house on a hill with a view. At this point the house might cost ten times the original amount.

There may also be many times during the project when you forgot to mention or thought of some other important feature, like an entertainment room, that you feel you must have! But your friend has already started working on the house and built the frame. And you might have to redo the entire design and knock down walls to accommodate, it which will make the project off budget, and add a lot of additional time.

It’s not always necessary to know everything up front, but it is important to anticipate large features so that your friend can design a plan that will accommodate changes later.

Steve McConnel, in his great book “Rapid Development”, gives a great summary of the software estimation story that all customers and managers must understand:

The basic software-estimation story is that software development is a process of gradual refinement. You begin with a fuzzy picture of what you want to build and then spend the rest of the project trying to bring that picture into clearer focus. Because the picture of the software you are trying to build is fuzzy, the estimate of the time and effort needed to build it is fuzzy, too. The estimate can come into focus only along with the software itself, which means that software-project estimation is also a process of gradual refinement.

In other words, it’s impossible for a programmer to estimate the project until he has a detailed understanding of what the customer wants. Until a specific feature is understood in detail, the rough estimate could be a factor of sixteen differences for that feature.

Another difficult part is that unless you’ve previously built the same house, you probably won’t have all the details, hence you have a fuzzy picture. Software development is a process of discovery and gradual refinement. Developers work with customers to unearth intricate business rules, features, and capabilities. One way you can bring a picture into the focus is by going through the process of defining a product, thinking through the requirements, and creating a design. Let’s go through what we need to have to better understand what we are trying to build.

Defining a product

Detailed understanding means that a developer can take the product definition and create a model of it in code. Most likely a developer is not a domain expert of the business and he needs a lot of help from the customer to understand it. Imagine that you get a job at an unfamiliar organization, where you had to learn how the business works before you can be productive. You will need a lot of time to get up to speed, and so will a developer in order to build your product. It’s also much cheaper to be make mistakes and have invalid assumptions at this stage, because no code is written and it doesn’t cost anything to correct it.

For the product owner at this stage, a good place to start is to define a clear problem/solution statement, along with user stories, tasks, and use cases. User stories is a simple description of a feature told in the following way:

As a <type of user>, I want <some goal> so that <some reason>.

For example, it may look like this: As an incentive compensation specialist I would be able to set up compensation wrappers so that I can create goals for payees. Does this story make a lot of sense to you? If you are not working in the incentive compensation business, I don’t think it will. Stories like these are an invitation for conversations between developer and business experts. During the conversation the developer will identify terms used, a vocabulary for that business domain, and know ubiquitous language in Domain Driven Design. It will also lead to a better understanding of a product, elements, subsystems and entities.

At this stage any additional information that a business expert can present would be helpful. Examples are screenshots of similar application, high level flowcharts, wireframes, onboarding manuals, etc. A developer can write a documentation summary about the problem/solution domain identifying all the entities, business vocabularies, definitions, subsystems, interactions and actors.

Requirements Specification

After defining the product and learning about the business, we can see a better picture, but it’s still not perfect; at this moment there could be a factor of eight differences between the high and low estimate. The next step is to zoom in even closer and define a list of requirements. The amount of features software can have has a tremendous impact on the design, time, and cost of the application. Even if you practice agile methodologies, you still need to gather a minimal set of requirements so you can plan for the future. Just like in the definition stage, it’s much easier, and cheaper, to make changes during requirements rather than realizing you’ve missed an important feature down the line, and have it take months to redesign the application. (Knock down walls).

Input at this stage from the business side should be a list of important features that this project must have. While a developer’s job is to understand how these features fit together and to create a plan that will accommodate future expansions.

Product Design

We are getting much closer to a better estimate at this point; the difference might be a factor of three between high and low estimates. Over my career I’ve learned that nothing brings software into a picture like creating a high level and detailed design. This is the stage where a developer takes everything learned about the product, and starts modeling the domain. During this stage she will make a lot of mistakes and flush out incorrect assumptions. Again, much cheaper to make mistakes here than during coding. Starting with a high level design of the major subsystems and defining a clear communication between them, then modeling your domain and define major objects and abstractions. (For a more detailed analysis on how to model a domain, see Eric Evan’s book, “Domain Driven Design”).

What if I don’t know what I want?

I think the best way to approach this problem is to hire a User Experience (UX) designer who can take vague ideas and create throw away sketches of the product. Each sketch can then be reviewed, refined, or redrawn. This process will flush out the requirements that will crystalize in an initial prototype, and can be tested on users before you start the expensive software development. A good UX designer in this situation will save you a lot of money because she will bring the requirements and prototype into focus, for the fraction of the cost of building software! By sketching, creating multiple prototypes, and then testing it on users, you know exactly what works and there is no chance of building something that your customers can’t use.

I see a tremendous amount of benefit of hiring a UX designer early in any software project. There is a lot of power in sketching, defining, and user testing multiple prototypes without actually building anything. It’s costly to experiment by building software and I don’t think anyone can get the user experience right the first time.

Getting an estimate

After the picture is more clear, a developer will create a list of low level details with a range estimate for how many man-hours it would take to complete each task. You must get estimates on low levels of details, this way you get a more accurate estimate and you can gauge how the project is progressing. A good developer will also provide additional information about the costs that are associated with each task and other alternatives that could be more cost effective. Like a good contractor who would present and educate you about different counter tops, their benefits, and costs, you want your developer to do the same for you when providing a list of estimates.

It’s also very important not to overlook other important tasks that are not coding, but have a significant impact on schedule and budget, like QA, documentation, deployments, setting up dev/production servers/databases, configuration, integration, holidays, training, and sick days. In addition there should be allowance with working with new languages, working with unfamiliar technologies, and frameworks. There is always a learning curve when doing something new and this should be accounted for.

Make estimation part of your plan Estimation should be part of your plan and you need to spend a significant amount of time on it; don’t rush it.

Word on Agile Software Development

I am a big believer in iterative development, rapid delivery, minimal planning, and many other principles Agile Manifesto stands for. Unfortunately, I’ve noticed some developers use Agile as an excuse to have no planning, design, or requirements analysis at all, which results in the costliest approach of them all: code-like-hell. It’s common, and you want to be wary of ballpark estimates with use of an Agile as an excuse to not do to any planning or design.