webserver - Nginx tries to proxy pass to upstream name -

hello have following quite simple configuration load balancer/fail over:

upstream backend_stream {     server 192.168.0.130:8080   max_fails=2 fail_timeout=30s;     server 192.168.0.131:8080   max_fails=2 fail_timeout=30s backup; }  server {     listen 443 ssl;     server_name exmaple.com;      # ssl stuff      proxy_ignore_client_abort on;     proxy_connect_timeout 3s;     proxy_read_timeout 5s;     proxy_send_timeout 5s;     send_timeout 20s;     proxy_next_upstream_timeout 60s;     proxy_next_upstream_tries 0;     proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504 non_idempotent;      proxy_pass  http://backend_stream;      access_log /path/to/access.log vhosts_extra;     error_log /path/to/error.log; } 

and following access log format:

log_format vhosts_extra '$host:$server_port $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" [$upstream_addr - $upstream_response_time] $request_time'; 

the important part @ end: [$upstream_addr - $upstream_response_time]. logging upstream server choosen , responses of upstream servers. nginx docs.

with such configuration wanted test traffic can handle. have used apache benchmark command line tool send number of concurent requests @ time, follow:

ab -t application/json -h 'authorization: basic xxx' -p ab_post.json -n 1000 -c 30 https://example.com 

and configuration handling requests time until starts choke (which ok, apps have limits). when first server starting overloaded , nginx has start using backup server strange happening. time time nginx trying pass request not upstream server, upstream name! of course not exist host. ... [192.168.0.131:8080, backend_stream - 5.000, 0.000] 5.051

see log fragment:

example.com:443 12.34.12.34 - - [17/nov/2016:16:15:54 +0000] "post / http/1.0" 200 1183 "-" "apachebench/2.3" [192.168.0.130:8080, 192.168.0.131:8080 - 5.000, 3.730] 8.783 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:54 +0000] "post / http/1.0" 200 1183 "-" "apachebench/2.3" [192.168.0.131:8080 - 3.843] 3.907 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:54 +0000] "post / http/1.0" 200 1183 "-" "apachebench/2.3" [192.168.0.130:8080, 192.168.0.131:8080 - 5.000, 3.942] 9.003 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:54 +0000] "post / http/1.0" 200 1183 "-" "apachebench/2.3" [192.168.0.131:8080 - 4.044] 4.106 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:55 +0000] "post / http/1.0" 200 1183 "-" "apachebench/2.3" [192.168.0.130:8080, 192.168.0.131:8080 - 5.000, 4.125] 9.182 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:55 +0000] "post / http/1.0" 200 1183 "-" "apachebench/2.3" [192.168.0.131:8080 - 4.249] 4.309 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:55 +0000] "post / http/1.0" 200 1183 "-" "apachebench/2.3" [192.168.0.130:8080, 192.168.0.131:8080 - 5.000, 4.306] 9.372 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:55 +0000] "post / http/1.0" 200 1183 "-" "apachebench/2.3" [192.168.0.131:8080 - 4.410] 4.477 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:55 +0000] "post / http/1.0" 200 1183 "-" "apachebench/2.3" [192.168.0.130:8080, 192.168.0.131:8080 - 5.001, 4.504] 9.566 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:56 +0000] "post / http/1.0" 200 1183 "-" "apachebench/2.3" [192.168.0.131:8080 - 4.583] 4.642 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:56 +0000] "post / http/1.0" 200 1183 "-" "apachebench/2.3" [192.168.0.130:8080, 192.168.0.131:8080 - 5.000, 4.707] 9.770 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:56 +0000] "post / http/1.0" 200 1183 "-" "apachebench/2.3" [192.168.0.131:8080 - 4.798] 4.860 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:56 +0000] "post / http/1.0" 200 1183 "-" "apachebench/2.3" [192.168.0.130:8080, 192.168.0.131:8080 - 5.000, 4.884] 9.945 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:56 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.000, 0.000] 5.048 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:57 +0000] "post / http/1.0" 504 176 "-" "apachebench/2.3" [192.168.0.130:8080, 192.168.0.131:8080 - 5.000, 5.002] 10.061 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:57 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.000, 0.000] 5.051 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:57 +0000] "post / http/1.0" 504 176 "-" "apachebench/2.3" [192.168.0.130:8080, 192.168.0.131:8080 - 5.001, 5.000] 10.059 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:57 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.001, 0.000] 5.052 

sometimes trying hard ...

example.com:443 12.34.12.34 - - [17/nov/2016:16:15:58 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.002, 0.000] 5.050 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:58 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.000, 0.000] 5.048 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:58 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.000, 0.000] 5.064 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:58 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.001, 0.000] 5.065 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:59 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.001, 0.000] 5.059 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:59 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.000, 0.000] 5.062 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:59 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.000, 0.000] 5.064 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:59 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.000, 0.000] 5.063 example.com:443 12.34.12.34 - - [17/nov/2016:16:15:59 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.000, 0.000] 5.066 example.com:443 12.34.12.34 - - [17/nov/2016:16:16:00 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.001, 0.000] 5.051 example.com:443 12.34.12.34 - - [17/nov/2016:16:16:00 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.000, 0.000] 5.055 example.com:443 12.34.12.34 - - [17/nov/2016:16:16:00 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.001, 0.000] 5.066 example.com:443 12.34.12.34 - - [17/nov/2016:16:16:00 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.000, 0.000] 5.057 example.com:443 12.34.12.34 - - [17/nov/2016:16:16:00 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.000, 0.000] 5.032 example.com:443 12.34.12.34 - - [17/nov/2016:16:16:01 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.001, 0.000] 5.001 example.com:443 12.34.12.34 - - [17/nov/2016:16:16:01 +0000] "post / http/1.0" 502 166 "-" "apachebench/2.3" [192.168.0.131:8080, backend_stream - 5.002, 0.000] 5.002 

my guess it happening when of upstream servers marked "failing", it's guess. cannot find such behaviour in docs or anywhere else.

is normal behavior , should aware may happen or doing wrong?

the nginx upstream enter link description heredocs stands:

if error occurs during communication server, request passed next server, , on until of functioning servers tried. if successful response not obtained of servers, client receive result of communication last server.

the worst part instant 502 bad gateway response.

some environment info:

nginx version: nginx/1.10.0 (ubuntu) 

nginx instaled default package repository extensions, nginx-extras package.

i had same problem you, found answer on this website here.

upstream name can seen in $upstream_addr variable if nginx wasn't able select upstream server connect because servers down per max_fails/fail_timeout. @ same time "no live upstreams" error logged error log.

i hope you.