Nginx

Published in

DevOps-Journey

11 min readJan 30, 2021

What should I know about it?

Nginx is a very popular web server, used as a simple web server, reverse proxy, caching for both API and content, and also for load balancing. I have been running Nginx machines for about 4 years now and I must admit one of the best web servers/ reverse proxy of all times. I believe some preface will help to understand this. Like many of my blogs reference links and credit, pages are added below

C10K problem

Around ten years ago, Daniel Kegel, a prominent software engineer, proclaimed that “it’s time for web servers to handle ten thousand clients simultaneously” and predicted what we now call Internet cloud services. Kegel’s C10K manifest spurred a number of attempts to solve the problem of web server optimization to handle a large number of clients at the same time, and Nginx turned out to be one of the most successful ones.

Some time in the early 90s Apache was the most used web server which was supporting many technology stacks. The initial approach on Apache was to keep spawning new worker process to create concurrency while it solved the concurrency problem, it still could not solve the issue of resource management and wastage. Let's take a minute to explain that

In order to provide concurrency typical approach is to create processes or threads. This approach moves each request to its own process/thread effectively providing the non-blocking IO behaviour.
The method listed above is effective as it is relatively easy to build
While it is relatively easy to build, it has a downside. Each process/worker needs resources to function. However small, it accumulates to a large number over time, considering the 90s hardware C10K problem.

Aimed at solving the C10K problem of 10,000 simultaneous connections, Nginx was written with a different architecture in mind — one which is much more suitable for nonlinear scalability in both the number of simultaneous connections and requests per second. nginx is event-based, so it does not follow Apache’s style of spawning new processes or threads for each web page request. The end result is that even as load increases, memory and CPU usage remain manageable. nginx can now deliver tens of thousands of concurrent connections on a server with typical hardware.

Nginx architecture

High-level architecture

Nginx worker architecture

Worker process handling

Request flow and context

Blocking IO vs NON-Blocking IO

Reference for devs

# Quick notesngx_events_module - NGX_CORE_MODULE- Create configuration to storing the related configuration- resolves : events{} :- static ngx_core_module_t ngx_events_module_ctx = {ngx_string("events"),NULL,ngx_event_init_conf};ngx_event_core_module - NGX_EVENT_MODULE- Store Basic configuration object of event module- Function - parse events{} and its sub-configuration## ngx_events_moduleModule definition:ngx_module_t ngx_events_module = {NGX_MODULE_V1,&ngx_events_module_ctx,                /* module context */ngx_events_commands,                   /* module directives */NGX_CORE_MODULE,                       /* module type */NULL,                                  /* init master */NULL,                                  /* init module */NULL,                                  /* init process */NULL,                                  /* init thread */NULL,                                  /* exit thread */NULL,                                  /* exit process */NULL,                                  /* exit master */NGX_MODULE_V1_PADDING};static ngx_core_module_t ngx_events_module_ctx = {   - Returns nginx core_module i.e. ngx_core_module_tngx_string("events"),NULL,                                            - No configuration defined for ngx_cycle_t  core object so usengx_event_init_conf                              - ngx_event_init_conf method to initialize the configuration};So where is the configuration object of the event module created here?In fact, it is carried out when parsing the configuration object,that is, in events{} the ngx_events_block() method of executing the parsing configuration block.This method essentially just creates an array of pointers,and then assigns it to the corresponding position of conf_ctx of ngx_cycle_t of nginx core and value object.static ngx_command_t ngx_events_commands[] = {{ngx_string("events"),                              - Configuration itemNGX_MAIN_CONF | NGX_CONF_BLOCK | NGX_CONF_NOARGS,  - NGX_CONF_BLOCKngx_events_block,                                  - Analysis is done via this method and assignment of core nginx coreconf_ctx and ngx_cycle_t value. Responsible to create array pointers.0,0,NULL},ngx_null_command};## ngx_event_core_module### Interface definitionstypedef struct {ngx_str_t *name;void *(*create_conf)(ngx_cycle_t *cycle);char *(*init_conf)(ngx_cycle_t *cycle, void *conf);ngx_event_actions_t actions;} ngx_event_module_t;### Methods for ngx_event_actions_the typical implementation interface of this interface is each event model defined by nginx,such as epoll ngx_epoll_module_ctx.actionsand kqueue ngx_kqueue_module_ctx.actions.It can be seen from this that the definition of the event module abstracts the relevant processingmethods of each model that handles events.typedef struct {# （epoll、kqueue)ngx_int_t (*add)(ngx_event_t *ev, ngx_int_t event, ngx_uint_t flags);ngx_int_t (*del)(ngx_event_t *ev, ngx_int_t event, ngx_uint_t flags);ngx_int_t (*enable)(ngx_event_t *ev, ngx_int_t event, ngx_uint_t flags);ngx_int_t (*disable)(ngx_event_t *ev, ngx_int_t event, ngx_uint_t flags);ngx_int_t (*add_conn)(ngx_connection_t *c);ngx_int_t (*del_conn)(ngx_connection_t *c, ngx_uint_t flags);ngx_int_t (*notify)(ngx_event_handler_pt handler);ngx_int_t (*process_events)(ngx_cycle_t *cycle, ngx_msec_t timer, ngx_uint_t flags);ngx_int_t (*init)(ngx_cycle_t *cycle, ngx_msec_t timer);void (*done)(ngx_cycle_t *cycle);} ngx_event_actions_t;                                              - ngx_event_actions_tit mainly defines the way the current event module handles each event.### Typengx_module_t ngx_event_core_module = {NGX_MODULE_V1,&ngx_event_core_module_ctx,            /* module context */     - Module contextngx_event_core_commands,               /* module directives */NGX_EVENT_MODULE,                      /* module type */NULL,                                  /* init master */ngx_event_module_init,                 /* init module */        - parser for event modulengx_event_process_init,                /* init process */       - called after nginx starts the worker process before the worker process starts to execute the main loop, and its role is to initialize the worker process before execution .NULL,                                  /* init thread */NULL,                                  /* exit thread */NULL,                                  /* exit process */NULL,                                  /* exit master */NGX_MODULE_V1_PADDING};static ngx_event_module_t ngx_event_core_module_ctx = {                 - Functions of this module&event_core_name,ngx_event_core_create_conf,            /* create configuration */ngx_event_core_init_conf,              /* init configuration */{NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL}        - Not responsible for processing of event handling schemes- Nginx needs this everywhere to be used by event handling modules for basic data and configuration};The ngx_event_core_modulethird attribute ngx_event_core_commands points to the third structure above.This structure defines the basic configuration of each configuration item that can be used by the current event moduleand the method of parsing these configuration items.static ngx_command_t ngx_event_core_commands[] = {{ngx_string("worker_connections"),NGX_EVENT_CONF | NGX_CONF_TAKE1,ngx_event_connections,0,0,NULL},{ngx_string("use"),NGX_EVENT_CONF | NGX_CONF_TAKE1,ngx_event_use,0,0,NULL},{ngx_string("multi_accept"),NGX_EVENT_CONF | NGX_CONF_FLAG,ngx_conf_set_flag_slot,0,offsetof(ngx_event_conf_t, multi_accept),NULL},{ngx_string("accept_mutex"),NGX_EVENT_CONF | NGX_CONF_FLAG,ngx_conf_set_flag_slot,0,offsetof(ngx_event_conf_t, accept_mutex),NULL},{ngx_string("accept_mutex_delay"),NGX_EVENT_CONF | NGX_CONF_TAKE1,ngx_conf_set_msec_slot,0,offsetof(ngx_event_conf_t, accept_mutex_delay),NULL},{ngx_string("debug_connection"),NGX_EVENT_CONF | NGX_CONF_TAKE1,ngx_event_debug_connection,0,0,NULL},ngx_null_command};### worker registrationstatic ngx_int_t ngx_event_process_init(ngx_cycle_t *cycle) {ngx_uint_t m, i;ngx_event_t *rev, *wev;ngx_listening_t *ls;ngx_connection_t *c, *next, *old;ngx_core_conf_t *ccf;ngx_event_conf_t *ecf;ngx_event_module_t *module;// Get Configuration Objects for Core Modulesccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_core_module);// Get the configuration object for the event core moduleecf = ngx_event_get_conf(cycle->conf_ctx, ngx_event_core_module);// Determine if three conditions are currently met, then mark the current way to use shared locks:// 1. Currently in master-worker mode;// 2. The number of current worker processes is greater than 1;// 3. The switch to use the shared lock is currently on;if (ccf->master && ccf->worker_processes > 1 && ecf->accept_mutex) {ngx_use_accept_mutex = 1;ngx_accept_mutex_held = 0;ngx_accept_mutex_delay = ecf->accept_mutex_delay;} else {// If the above conditions are not met, specify that shared locks are not usedngx_use_accept_mutex = 0;}#if (NGX_WIN32)/** disable accept mutex on win32 as it may cause deadlock if* grabbed by a process which can't accept connections*/ngx_use_accept_mutex = 0;#endif// The main function of these two queues here is that each worker process receives client accept events after it acquires a shared lock.// It is then placed in the ngx_posted_accept_events queue, events in the queue are processed, and client connections are added to the queue// In the ngx_posted_events queue, and then release the lock, that is, the worker process that acquires the lock only needs an accept client connection.// The lock is then given permission to other processes, and the read and write events for the received connections are handled on their own.// Create the ngx_posted_accept_events queue used to receive connection events from clientsngx_queue_init(&ngx_posted_accept_events);// Create the ngx_posted_events queue that handles read and write events for client connectionsngx_queue_init(&ngx_posted_events);// Initialize a red-black tree to store eventsif (ngx_event_timer_init(cycle->log) == NGX_ERROR) {return NGX_ERROR;}for (m = 0; cycle->modules[m]; m++) {if (cycle->modules[m]->type != NGX_EVENT_MODULE) {continue;}// Ecf->use stores the module number of the selected event model, which is found hereif (cycle->modules[m]->ctx_index != ecf->use) {continue;}// Module is the module corresponding to the selected event modelmodule = cycle->modules[m]->ctx;// Initialization method invoked for specified event modelif (module->actions.init(cycle, ngx_timer_resolution) != NGX_OK) {/* fatal */exit(2);}break;}#if !(NGX_WIN32)// ngx_timer_resolution represents the time interval between sending update time events// This means if ngx_timer_resolution is set and no timer events are set.// ngx_event_flags is set in the initialization of the event module, and only the eventport and kqueue models will// NGX_USE_TIMER_EVENT is set to ngx_event_flagsif (ngx_timer_resolution && !(ngx_event_flags & NGX_USE_TIMER_EVENT)) {struct sigaction sa;struct itimerval itv;ngx_memzero(&sa, sizeof(struct sigaction));// The sa here is mainly to add the following SIGALRM signal listening events, which are sent to the current process at regular intervals// When the current process receives a signal, the following ngx_timer_signal_handler() method will be called, which will// ngx_event_timer_alarm is set to 1, and when the current process is in an event loop, determine if// ngx_event_timer_alarm is 1, which updates the time data cached by the current processsa.sa_handler = ngx_timer_signal_handler;sigemptyset(&sa.sa_mask);// Add SIGALRM listening signalif (sigaction(SIGALRM, &sa, NULL) == -1) {ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,"sigaction(SIGALRM) failed");return NGX_ERROR;}// Set time interval related parametersitv.it_interval.tv_sec = ngx_timer_resolution / 1000;itv.it_interval.tv_usec = (ngx_timer_resolution % 1000) * 1000;itv.it_value.tv_sec = ngx_timer_resolution / 1000;itv.it_value.tv_usec = (ngx_timer_resolution % 1000) * 1000;// Set timer at specified intervalif (setitimer(ITIMER_REAL, &itv, NULL) == -1) {ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,"setitimer() failed");}}// NGX_USE_FD_EVENT indicates that the event filter has no transparent data and requires a file descriptor table, which is mainly used for poll, /dev/pollif (ngx_event_flags & NGX_USE_FD_EVENT) {struct rlimit rlmt;if (getrlimit(RLIMIT_NOFILE, &rlmt) == -1) {ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,"getrlimit(RLIMIT_NOFILE) failed");return NGX_ERROR;}// The main thing here is to initialize the maximum number of ngx_connection_t structures and save them in the files arraycycle->files_n = (ngx_uint_t) rlmt.rlim_cur;cycle->files = ngx_calloc(sizeof(ngx_connection_t *) * cycle->files_n, cycle->log);if (cycle->files == NULL) {return NGX_ERROR;}}#elseif (ngx_timer_resolution && !(ngx_event_flags & NGX_USE_TIMER_EVENT)) {ngx_log_error(NGX_LOG_WARN, cycle->log, 0,"the \"timer_resolution\" directive is not supported ""with the configured event method, ignored");ngx_timer_resolution = 0;}#endif// Request a specified number of ngx_connection_t arrays, where connection_n corresponds to configuration// Size specified by worker_connections in filecycle->connections = ngx_alloc(sizeof(ngx_connection_t) * cycle->connection_n, cycle->log);if (cycle->connections == NULL) {return NGX_ERROR;}c = cycle->connections;// Request the specified number of ngx_event_t arrays, which are the same length as the connections array.// This allows you to correlate the connections array with the read_events arraycycle->read_events = ngx_alloc(sizeof(ngx_event_t) * cycle->connection_n, cycle->log);if (cycle->read_events == NULL) {return NGX_ERROR;}rev = cycle->read_events;for (i = 0; i < cycle->connection_n; i++) {rev[i].closed = 1;  // Initial state Default read events are closed staterev[i].instance = 1;  // Initialize instance1 initially}// Request the specified number of ngx_event_t arrays, which are the same length as the connections array.// This allows you to correlate the connections array with the write_events arraycycle->write_events = ngx_alloc(sizeof(ngx_event_t) * cycle->connection_n, cycle->log);if (cycle->write_events == NULL) {return NGX_ERROR;}wev = cycle->write_events;for (i = 0; i < cycle->connection_n; i++) {wev[i].closed = 1;  // Write events initially are also closed by default}i = cycle->connection_n;next = NULL;do {i--;// The elements of the read_events and write_events arrays are assigned successively to the read and write attributes of the elements of the connections array.// And assemble the connections array into a single-chain tablec[i].data = next;c[i].read = &cycle->read_events[i];c[i].write = &cycle->write_events[i];c[i].fd = (ngx_socket_t) -1;next = &c[i];} while (i);// At the initial state, all connections are unused and need to be stored in the free_connections chain tablecycle->free_connections = next;cycle->free_connection_n = cycle->connection_n;/* for each listening socket */ls = cycle->listening.elts;for (i = 0; i < cycle->listening.nelts; i++) {#if (NGX_HAVE_REUSEPORT)if (ls[i].reuseport && ls[i].worker != ngx_worker) {continue;}#endif// Here is a ngx_connection_t structure bound to each port currently being listened onc = ngx_get_connection(ls[i].fd, cycle->log);if (c == NULL) {return NGX_ERROR;}c->type = ls[i].type;c->log = &ls[i].log;c->listening = &ls[i];ls[i].connection = c;rev = c->read;rev->log = c->log;// Mark accept as 1 to indicate that client connection events are currently available for receptionrev->accept = 1;#if (NGX_HAVE_DEFERRED_ACCEPT)rev->deferred_accept = ls[i].deferred_accept;#endifif (!(ngx_event_flags & NGX_USE_IOCP_EVENT)) {if (ls[i].previous) {/** delete the old accept events that were bound to* the old cycle read events array*/// Delete Old Eventsold = ls[i].previous->connection;if (ngx_del_event(old->read, NGX_READ_EVENT, NGX_CLOSE_EVENT) == NGX_ERROR) {return NGX_ERROR;}old->fd = (ngx_socket_t) -1;}}#if (NGX_WIN32)if (ngx_event_flags & NGX_USE_IOCP_EVENT) {ngx_iocp_conf_t  *iocpcf;rev->handler = ngx_event_acceptex;if (ngx_use_accept_mutex) {continue;}if (ngx_add_event(rev, 0, NGX_IOCP_ACCEPT) == NGX_ERROR) {return NGX_ERROR;}ls[i].log.handler = ngx_acceptex_log_error;iocpcf = ngx_event_get_conf(cycle->conf_ctx, ngx_iocp_module);if (ngx_event_post_acceptex(&ls[i], iocpcf->post_acceptex)== NGX_ERROR){return NGX_ERROR;}} else {rev->handler = ngx_event_accept;if (ngx_use_accept_mutex) {continue;}if (ngx_add_event(rev, NGX_READ_EVENT, 0) == NGX_ERROR) {return NGX_ERROR;}}#else// SOCK_STREAM represents TCP, which is generally TCP, i.e. after receiving an accept event from a client.// The ngx_event_accept() method is called to handle the eventrev->handler = (c->type == SOCK_STREAM) ? ngx_event_accept: ngx_event_recvmsg;#if (NGX_HAVE_REUSEPORT)// Add current event to event listening queueif (ls[i].reuseport) {if (ngx_add_event(rev, NGX_READ_EVENT, 0) == NGX_ERROR) {return NGX_ERROR;}continue;}#endifif (ngx_use_accept_mutex) {continue;}#if (NGX_HAVE_EPOLLEXCLUSIVE)if ((ngx_event_flags & NGX_USE_EPOLL_EVENT)&& ccf->worker_processes > 1){if (ngx_add_event(rev, NGX_READ_EVENT, NGX_EXCLUSIVE_EVENT)== NGX_ERROR){return NGX_ERROR;}continue;}#endif// Add current event to event listening queueif (ngx_add_event(rev, NGX_READ_EVENT, 0) == NGX_ERROR) {return NGX_ERROR;}#endif}return NGX_OK;}