• 6
name

A PHP Error was encountered

Severity: Notice

Message: Undefined index: userid

Filename: views/question.php

Line Number: 191

Backtrace:

File: /home/prodcxja/public_html/questions/application/views/question.php
Line: 191
Function: _error_handler

File: /home/prodcxja/public_html/questions/application/controllers/Questions.php
Line: 433
Function: view

File: /home/prodcxja/public_html/questions/index.php
Line: 315
Function: require_once

I am running redis-server (ver. 2.8.4) on Ubuntu 14.04 in Rackspace Cloud Server. The redis-server process is being run by the redis user.

Early this morning, the Redis server spontaneously began failing "opening .rdb for saving." This occurred after the process had been up for over 20 days without incident.

As you can see in the snippet from the logs below, Redis was behaving normally, and then started failing around at 8:08am UTC. Power-cycling the server later resolved the issue.

[26083] 31 Aug 07:54:47.089 * RDB: 2 MB of memory used by copy-on-write
[17183] 31 Aug 07:54:47.186 * Background saving terminated with success
[17183] 31 Aug 07:59:48.028 * 10 changes in 300 seconds. Saving...
[17183] 31 Aug 07:59:48.029 * Background saving started by pid 26653
[26653] 31 Aug 07:59:48.032 * DB saved on disk
[26653] 31 Aug 07:59:48.032 * RDB: 2 MB of memory used by copy-on-write
[17183] 31 Aug 07:59:48.129 * Background saving terminated with success
[17183] 31 Aug 08:04:49.061 * 10 changes in 300 seconds. Saving...
[17183] 31 Aug 08:04:49.061 * Background saving started by pid 27197
[27197] 31 Aug 08:04:49.064 * DB saved on disk
[27197] 31 Aug 08:04:49.065 * RDB: 2 MB of memory used by copy-on-write
[17183] 31 Aug 08:04:49.162 * Background saving terminated with success
[17183] 31 Aug 08:07:56.881 * DB saved on disk
[17183] 31 Aug 08:08:05.880 # Failed opening .rdb for saving: Permission denied
[17183] 31 Aug 08:12:57.047 * 10 changes in 300 seconds. Saving...
[17183] 31 Aug 08:12:57.047 * Background saving started by pid 28087
[28087] 31 Aug 08:12:57.048 # Failed opening .rdb for saving: Permission denied
[17183] 31 Aug 08:12:57.148 # Background saving error
[17183] 31 Aug 08:13:03.068 * 10 changes in 300 seconds. Saving...
[17183] 31 Aug 08:13:03.069 * Background saving started by pid 28088
[28088] 31 Aug 08:13:03.070 # Failed opening .rdb for saving: Permission denied
[17183] 31 Aug 08:13:03.169 # Background saving error
[17183] 31 Aug 08:13:09.087 * 10 changes in 300 seconds. Saving...
[17183] 31 Aug 08:13:09.087 * Background saving started by pid 28089
[28089] 31 Aug 08:13:09.088 # Failed opening .rdb for saving: Permission denied
(... continues like this until server restart)

So, my question: how could this happen?

I'm stumped, but here are my vague hunches:

  • Memory use too high (I am running just a 512MB instance and Redis has warned me to set 'vm.overcommit_memory = 1' in /etc/sysctl.conf)
  • Redis hit some kind of internal limit after 20 days of continual operation (and so this will happen again in another ~20 days).
  • Rackspace did something at 8am UTC (4am Eastern time) that somehow impacted my server.
      • 1
    • When I see errors about permission denied, I check permissions. How could it happen? You need to figure out WHAT happened before you can figure out how or why it happened.

Run redis-cli and look at the output for CONFIG GET dbfilename and CONFIG GET dir

I have a feeling those may have been changed. Values should be dump.rdb and something like /var/lib/redis, respectively.

If they have changed, someone compromised your redis server. If that's the case, check to see if your redis service is open to the public. You shouldn't be able to connect to it from an external IP.

  • 11
Reply Report
      • 2
    • I believe this is the right answer in this case. I was coming to the same conclusion based on discussions like this: meta.discourse.org/t/losing-connection-with-redis/36343 -- It seems Redis can be remotely exploited to make temporary changes that revert on restart. Of course, it could also be exploited to do worse than that. Port 6379 was indeed open to the internet at the time this happened, and it never should have been. Thanks for your help!

Warm tip !!!

This article is reproduced from Stack Exchange / Stack Overflow, please click

Trending Tags