| 1 | issuekey | title | description | storypoint |
|---|
| 2 | MESOS-313 | Report executor terminations to framework schedulers. | The Scheduler interface has a callback for executorLost, but currently it is never called. | 2 |
| 3 | MESOS-336 | Mesos slave should cache executors | The slave should be smarter about how it handles pulling down executors. In our environment, executors rarely change but the slave will always pull it down from regardless HDFS. This puts undue stress on our HDFS clusters, and is not resilient to reduced HDFS availability. | 5 |
| 4 | MESOS-343 | Expose TASK_FAILED reason to Frameworks. | We now have a message string inside TaskStatus that provides human readable information about TASK_FAILED. It would be good to add some structure to the failure reasons, for framework schedulers to act on programmatically. E.g. enum TaskFailure { EXECUTOR_OOM; EXECUTOR_OUT_OF_DISK; EXECUTOR_TERMINATED; SLAVE_LOST; etc.. } | 8 |
| 5 | MESOS-487 | Balloon framework fails to run due to bad flags | I suspect this has to do with the latest flags refactor. [vinod@smfd-bkq-03-sr4 build]$ sudo GLOG_v=1 ./bin/mesos-tests.sh --gtest_filter="*Balloon*" --verbose WARNING: Logging before InitGoogleLogging() is written to STDERR I0529 22:28:13.094351 31506 process.cpp:1426] libprocess is initialized on 10.37.184.103:53425 for 24 cpus I0529 22:28:13.095010 31506 logging.cpp:91] Logging to STDERR Source directory: /home/vinod/mesos Build directory: /home/vinod/mesos/build ------------------------------------------------------------- We cannot run any cgroups tests that require mounting hierarchies because you have the following hierarchies mounted: /cgroup We'll disable the CgroupsNoHierarchyTest test fixture for now. ------------------------------------------------------------- Note: Google Test filter = *Balloon*-CgroupsNoHierarchyTest.ROOT_CGROUPS_NOHIERARCHY_MountUnmountHierarchy: [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from CgroupsIsolatorTest [ RUN ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework Using temporary directory '/tmp/CgroupsIsolatorTest_ROOT_CGROUPS_BalloonFramework_pWWdE1' Launched master at 31574 Failed to load unknown flag 'build_dir' Usage: lt-mesos-master [...] Supported options: --allocation_interval=VALUE Amount of time to wait between performing (batch) allocations (e.g., 500ms, 1sec, etc) (default: 1secs) --cluster=VALUE Human readable name for the cluster, displayed in the webui --framework_sorter=VALUE Policy to use for allocating resources between a given user's frameworks. Options are the same as for user_allocator (default: drf) --[no-]help Prints this help message (default: false) --ip=VALUE IP address to listen on --log_dir=VALUE Location to put log files (no default, nothing is written to disk unless specified; does not affect logging to stderr) --logbufsecs=VALUE How many seconds to buffer log messages for (default: 0) --port=VALUE Port to listen on (default: 5050) --[no-]quiet Disable logging to stderr (default: false) --[no-]root_submissions Can root submit frameworks? (default: true) --slaves=VALUE Initial slaves that should be considered part of this cluster (or if using ZooKeeper a URL) (default: *) --user_sorter=VALUE Policy to use for allocating resources between users. May be one of: dominant_resource_fairness (drf) (default: drf) --webui_dir=VALUE Location of the webui files/assets (default: /usr/local/share/mesos/webui) --whitelist=VALUE Path to a file with a list of slaves (one per line) to advertise offers for; should be of the form: file://path/to/file (default: *) --zk=VALUE ZooKeeper URL (used for leader election amongst masters) May be one of: zk://host1:port1,host2:port2,.../path zk://username:password@host1:port1,host2:port2,.../path file://path/to/file (where file contains one of the above) (default: ) {RED}Master crashed; failing test /home/vinod/mesos/src/tests/balloon_framework_test.sh: line 31: kill: (31574) - No such process ../../src/tests/script.cpp:76: Failure Failed balloon_framework_test.sh exited with status 2 [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework (2031 ms) [----------] 1 test from CgroupsIsolatorTest (2031 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (2031 ms total) [ PASSED ] 0 tests. [ FAILED ] 1 test, listed below: [ FAILED ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework 1 FAILED TEST | 1 |
| 6 | MESOS-598 | Also check 'git diff --shortstat --staged' in post-reviews.py. | We current check if you have any changes before we run post-reviews.py but we don't check for staged changes which IIUC could get lost. | 1 |
| 7 | MESOS-708 | Static files missing "Last-Modified" HTTP headers | Static assets served by the Mesos master don't return "Last-Modified" HTTP headers. That means clients receive a 200 status code and re-download assets on every page request even if the assets haven't changed. Because Angular JS does most of the work, the downloading happens only when you navigate to Mesos master in your browser or use the browser's refresh. Example header for "mesos.css": HTTP/1.1 200 OK Date: Thu, 26 Sep 2013 17:18:52 GMT Content-Length: 1670 Content-Type: text/css Clients sometimes use the "Date" header for the same effect as "Last-Modified", but the date is always the time of the response from the server, i.e. it changes on every request and makes the assets look new every time. The "Last-Modified" header should be added and should be the last modified time of the file. On subsequent requests for the same files, the master should return 304 responses with no content rather than 200 with the full files. It could save clients a lot of download time since Mesos assets are rather heavyweight. | 2 |
| 8 | MESOS-723 | Expose total number of resources allocated to the slave in its endpoint | This could be useful information if there are bugs in master/slave that causes slaves to overcommit its resources. | 2 |
| 9 | MESOS-752 | SlaveRecoveryTest/0.ReconcileTasksMissingFromSlave test is flaky | [ RUN ] SlaveRecoveryTest/0.ReconcileTasksMissingFromSlave Checkpointing executor's forked pid 32281 to '/tmp/SlaveRecoveryTest_0_ReconcileTasksMissingFromSlave_NT1btb/meta/slaves/201310151913-16777343-35153-31491-0/frameworks/201310151913-16777343-35153-31491-0000/executors/0514b52f-3c17-4ee5-ba16-635198701ca2/runs/97c9e2cc-ceea-40a8-a915-aed5fed1dcb3/pids/forked.pid' Fetching resources into '/tmp/SlaveRecoveryTest_0_ReconcileTasksMissingFromSlave_NT1btb/slaves/201310151913-16777343-35153-31491-0/frameworks/201310151913-16777343-35153-31491-0000/executors/0514b52f-3c17-4ee5-ba16-635198701ca2/runs/97c9e2cc-ceea-40a8-a915-aed5fed1dcb3' Registered executor on localhost.localdomain Starting task 0514b52f-3c17-4ee5-ba16-635198701ca2 Forked command at 32317 sh -c 'sleep 10' tests/slave_recovery_tests.cpp:1927: Failure Mock function called more times than expected - returning directly. Function call: statusUpdate(0x7fffae636eb0, @0x7f1590027a00 64-byte object <F0-2F D0-A1 15-7F 00-00 00-00 00-00 00-00 00-00 40-E9 01-90 15-7F 00-00 20-6B 03-90 15-7F 00-00 48-91 C3-00 00-00 00-00 B0-3B 01-90 15-7F 00-00 05-00 00-00 00-00 00-00 17-00 00-00 00-00 00-00>) Expected: to be called once Actual: called twice - over-saturated and active Command exited with status 0 (pid: 32317) | 1 |
| 10 | MESOS-786 | Update semantics of when framework registered()/reregistered() get called | Current semantics: 1) Framework connects w/ master very first time --> registered() 2) Framework reconnects w/ same master after a zk blip --> reregistered() 3) Framework reconnects w/ failed over master --> registered() 4) Failed over framework connects w/ same master --> registered() 5) Failed over framework connects w/ failed over master --> registered() Updated semantics: Everything same except 3) Framework reconnects w/ failed over master --> reregistered() | 3 |
| 11 | MESOS-830 | ExamplesTest.JavaFramework is flaky | Identify the cause of the following test failure: [ RUN ] ExamplesTest.JavaFramework Using temporary directory '/tmp/ExamplesTest_JavaFramework_wSc7u8' Enabling authentication for the framework I1120 15:13:39.820032 1681264640 master.cpp:285] Master started on 172.25.133.171:52576 I1120 15:13:39.820180 1681264640 master.cpp:299] Master ID: 201311201513-2877626796-52576-3234 I1120 15:13:39.820194 1681264640 master.cpp:302] Master only allowing authenticated frameworks to register! I1120 15:13:39.821197 1679654912 slave.cpp:112] Slave started on 1)@172.25.133.171:52576 I1120 15:13:39.821795 1679654912 slave.cpp:212] Slave resources: cpus(*):4; mem(*):7168; disk(*):481998; ports(*):[31000-32000] I1120 15:13:39.822855 1682337792 slave.cpp:112] Slave started on 2)@172.25.133.171:52576 I1120 15:13:39.823652 1682337792 slave.cpp:212] Slave resources: cpus(*):4; mem(*):7168; disk(*):481998; ports(*):[31000-32000] I1120 15:13:39.825330 1679118336 master.cpp:744] The newly elected leader is master@172.25.133.171:52576 I1120 15:13:39.825445 1679118336 master.cpp:748] Elected as the leading master! I1120 15:13:39.825907 1681264640 state.cpp:33] Recovering state from '/tmp/ExamplesTest_JavaFramework_wSc7u8/0/meta' I1120 15:13:39.826127 1681264640 status_update_manager.cpp:180] Recovering status update manager I1120 15:13:39.826331 1681801216 process_isolator.cpp:317] Recovering isolator I1120 15:13:39.826738 1682874368 slave.cpp:2743] Finished recovery I1120 15:13:39.827747 1682337792 state.cpp:33] Recovering state from '/tmp/ExamplesTest_JavaFramework_wSc7u8/1/meta' I1120 15:13:39.827945 1680191488 slave.cpp:112] Slave started on 3)@172.25.133.171:52576 I1120 15:13:39.828415 1682337792 status_update_manager.cpp:180] Recovering status update manager I1120 15:13:39.828608 1680728064 sched.cpp:260] Authenticating with master master@172.25.133.171:52576 I1120 15:13:39.828606 1680191488 slave.cpp:212] Slave resources: cpus(*):4; mem(*):7168; disk(*):481998; ports(*):[31000-32000] I1120 15:13:39.828680 1682874368 slave.cpp:497] New master detected at master@172.25.133.171:52576 I1120 15:13:39.828765 1682337792 process_isolator.cpp:317] Recovering isolator I1120 15:13:39.829828 1680728064 sched.cpp:229] Detecting new master I1120 15:13:39.830288 1679654912 authenticatee.hpp:100] Initializing client SASL I1120 15:13:39.831635 1680191488 state.cpp:33] Recovering state from '/tmp/ExamplesTest_JavaFramework_wSc7u8/2/meta' I1120 15:13:39.831991 1679118336 status_update_manager.cpp:158] New master detected at master@172.25.133.171:52576 I1120 15:13:39.832042 1682874368 slave.cpp:524] Detecting new master I1120 15:13:39.832314 1682337792 slave.cpp:2743] Finished recovery I1120 15:13:39.832309 1681264640 master.cpp:1266] Attempting to register slave on vkone.local at slave(1)@172.25.133.171:52576 I1120 15:13:39.832929 1680728064 status_update_manager.cpp:180] Recovering status update manager I1120 15:13:39.833371 1681801216 slave.cpp:497] New master detected at master@172.25.133.171:52576 I1120 15:13:39.833273 1681264640 master.cpp:2513] Adding slave 201311201513-2877626796-52576-3234-0 at vkone.local with cpus(*):4; mem(*):7168; disk(*):481998; ports(*):[31000-32000] I1120 15:13:39.833595 1680728064 process_isolator.cpp:317] Recovering isolator I1120 15:13:39.833859 1681801216 slave.cpp:524] Detecting new master I1120 15:13:39.833861 1682874368 status_update_manager.cpp:158] New master detected at master@172.25.133.171:52576 I1120 15:13:39.834092 1680191488 slave.cpp:542] Registered with master master@172.25.133.171:52576; given slave ID 201311201513-2877626796-52576-3234-0 I1120 15:13:39.834486 1681264640 master.cpp:1266] Attempting to register slave on vkone.local at slave(2)@172.25.133.171:52576 I1120 15:13:39.834549 1681264640 master.cpp:2513] Adding slave 201311201513-2877626796-52576-3234-1 at vkone.local with cpus(*):4; mem(*):7168; disk(*):481998; ports(*):[31000-32000] I1120 15:13:39.834750 1680191488 slave.cpp:555] Checkpointing SlaveInfo to '/tmp/ExamplesTest_JavaFramework_wSc7u8/0/meta/slaves/201311201513-2877626796-52576-3234-0/slave.info' I1120 15:13:39.834875 1682874368 hierarchical_allocator_process.hpp:445] Added slave 201311201513-2877626796-52576-3234-0 (vkone.local) with cpus(*):4; mem(*):7168; disk(*):481998; ports(*):[31000-32000] (and cpus(*):4; mem(*):7168; disk(*):481998; ports(*):[31000-32000] available) I1120 15:13:39.835155 1680728064 slave.cpp:542] Registered with master master@172.25.133.171:52576; given slave ID 201311201513-2877626796-52576-3234-1 I1120 15:13:39.835458 1679118336 slave.cpp:2743] Finished recovery I1120 15:13:39.835739 1680728064 slave.cpp:555] Checkpointing SlaveInfo to '/tmp/ExamplesTest_JavaFramework_wSc7u8/1/meta/slaves/201311201513-2877626796-52576-3234-1/slave.info' I1120 15:13:39.835922 1682874368 hierarchical_allocator_process.hpp:445] Added slave 201311201513-2877626796-52576-3234-1 (vkone.local) with cpus(*):4; mem(*):7168; disk(*):481998; ports(*):[31000-32000] (and cpus(*):4; mem(*):7168; disk(*):481998; ports(*):[31000-32000] available) I1120 15:13:39.836120 1681264640 slave.cpp:497] New master detected at master@172.25.133.171:52576 I1120 15:13:39.836340 1679118336 status_update_manager.cpp:158] New master detected at master@172.25.133.171:52576 I1120 15:13:39.836436 1681264640 slave.cpp:524] Detecting new master I1120 15:13:39.836629 1682874368 master.cpp:1266] Attempting to register slave on vkone.local at slave(3)@172.25.133.171:52576 I1120 15:13:39.836653 1682874368 master.cpp:2513] Adding slave 201311201513-2877626796-52576-3234-2 at vkone.local with cpus(*):4; mem(*):7168; disk(*):481998; ports(*):[31000-32000] I1120 15:13:39.836804 1680728064 slave.cpp:542] Registered with master master@172.25.133.171:52576; given slave ID 201311201513-2877626796-52576-3234-2 I1120 15:13:39.837190 1680728064 slave.cpp:555] Checkpointing SlaveInfo to '/tmp/ExamplesTest_JavaFramework_wSc7u8/2/meta/slaves/201311201513-2877626796-52576-3234-2/slave.info' I1120 15:13:39.837569 1682874368 hierarchical_allocator_process.hpp:445] Added slave 201311201513-2877626796-52576-3234-2 (vkone.local) with cpus(*):4; mem(*):7168; disk(*):481998; ports(*):[31000-32000] (and cpus(*):4; mem(*):7168; disk(*):481998; ports(*):[31000-32000] available) I1120 15:13:39.852011 1679654912 authenticatee.hpp:124] Creating new client SASL connection I1120 15:13:39.852219 1680191488 master.cpp:1734] Authenticating framework at scheduler(1)@172.25.133.171:52576 I1120 15:13:39.852577 1682337792 authenticator.hpp:83] Initializing server SASL I1120 15:13:39.856160 1682337792 authenticator.hpp:140] Creating new server SASL connection I1120 15:13:39.856334 1681264640 authenticatee.hpp:212] Received SASL authentication mechanisms: CRAM-MD5 I1120 15:13:39.856360 1681264640 authenticatee.hpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I1120 15:13:39.856421 1681264640 authenticator.hpp:243] Received SASL authentication start I1120 15:13:39.856487 1681264640 authenticator.hpp:325] Authentication requires more steps I1120 15:13:39.856531 1681264640 authenticatee.hpp:258] Received SASL authentication step I1120 15:13:39.856576 1681264640 authenticator.hpp:271] Received SASL authentication step I1120 15:13:39.856643 1681264640 authenticator.hpp:317] Authentication success I1120 15:13:39.856724 1681264640 authenticatee.hpp:298] Authentication success I1120 15:13:39.856768 1681264640 master.cpp:1774] Successfully authenticated framework at scheduler(1)@172.25.133.171:52576 I1120 15:13:39.857028 1681264640 sched.cpp:334] Successfully authenticated with master master@172.25.133.171:52576 I1120 15:13:39.857139 1681264640 master.cpp:798] Received registration request from scheduler(1)@172.25.133.171:52576 I1120 15:13:39.857306 1681264640 master.cpp:816] Registering framework 201311201513-2877626796-52576-3234-0000 at scheduler(1)@172.25.133.171:52576 I1120 15:13:39.862296 1680191488 hierarchical_allocator_process.hpp:332] Added framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:39.863867 1680191488 master.cpp:1700] Sending 3 offers to framework 201311201513-2877626796-52576-3234-0000 Registered! ID = 201311201513-2877626796-52576-3234-0000 Launching task 0 Launching task 1 Launching task 2 I1120 15:13:39.905390 1680191488 master.cpp:2026] Processing reply for offer 201311201513-2877626796-52576-3234-0 on slave 201311201513-2877626796-52576-3234-1 (vkone.local) for framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:39.905825 1680191488 master.hpp:400] Adding task 0 with resources cpus(*):1; mem(*):128 on slave 201311201513-2877626796-52576-3234-1 (vkone.local) I1120 15:13:39.905886 1680191488 master.cpp:2150] Launching task 0 of framework 201311201513-2877626796-52576-3234-0000 with resources cpus(*):1; mem(*):128 on slave 201311201513-2877626796-52576-3234-1 (vkone.local) I1120 15:13:39.906422 1680191488 master.cpp:2026] Processing reply for offer 201311201513-2877626796-52576-3234-1 on slave 201311201513-2877626796-52576-3234-2 (vkone.local) for framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:39.906664 1680191488 master.hpp:400] Adding task 1 with resources cpus(*):1; mem(*):128 on slave 201311201513-2877626796-52576-3234-2 (vkone.local) I1120 15:13:39.906721 1680191488 master.cpp:2150] Launching task 1 of framework 201311201513-2877626796-52576-3234-0000 with resources cpus(*):1; mem(*):128 on slave 201311201513-2877626796-52576-3234-2 (vkone.local) I1120 15:13:39.907171 1680191488 master.cpp:2026] Processing reply for offer 201311201513-2877626796-52576-3234-2 on slave 201311201513-2877626796-52576-3234-0 (vkone.local) for framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:39.907419 1680191488 master.hpp:400] Adding task 2 with resources cpus(*):1; mem(*):128 on slave 201311201513-2877626796-52576-3234-0 (vkone.local) I1120 15:13:39.907480 1680191488 master.cpp:2150] Launching task 2 of framework 201311201513-2877626796-52576-3234-0000 with resources cpus(*):1; mem(*):128 on slave 201311201513-2877626796-52576-3234-0 (vkone.local) I1120 15:13:39.907938 1680191488 slave.cpp:722] Got assigned task 0 for framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:39.908473 1680191488 slave.cpp:833] Launching task 0 for framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:39.914427 1682874368 slave.cpp:722] Got assigned task 1 for framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:39.914594 1680728064 slave.cpp:722] Got assigned task 2 for framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:39.914844 1681801216 hierarchical_allocator_process.hpp:590] Framework 201311201513-2877626796-52576-3234-0000 filtered slave 201311201513-2877626796-52576-3234-1 for 1secs I1120 15:13:39.915292 1682874368 slave.cpp:833] Launching task 1 for framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:39.915424 1681801216 hierarchical_allocator_process.hpp:590] Framework 201311201513-2877626796-52576-3234-0000 filtered slave 201311201513-2877626796-52576-3234-2 for 1secs I1120 15:13:39.915685 1681801216 hierarchical_allocator_process.hpp:590] Framework 201311201513-2877626796-52576-3234-0000 filtered slave 201311201513-2877626796-52576-3234-0 for 1secs I1120 15:13:39.915828 1680728064 slave.cpp:833] Launching task 2 for framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:39.917840 1680191488 slave.cpp:943] Queuing task '0' for executor default of framework '201311201513-2877626796-52576-3234-0000 I1120 15:13:39.917935 1679118336 process_isolator.cpp:100] Launching default (/Users/vinod/workspace/apache/mesos/build/src/examples/java/test-executor) in /tmp/ExamplesTest_JavaFramework_wSc7u8/1/slaves/201311201513-2877626796-52576-3234-1/frameworks/201311201513-2877626796-52576-3234-0000/executors/default/runs/375b31a9-7093-4db1-964d-e6b425b1e4b4 with resources ' for framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:39.922019 1679118336 process_isolator.cpp:163] Forked executor at 3268 I1120 15:13:39.922703 1679118336 slave.cpp:2073] Monitoring executor default of framework 201311201513-2877626796-52576-3234-0000 forked at pid 3268 I1120 15:13:39.929134 1682874368 slave.cpp:943] Queuing task '1' for executor default of framework '201311201513-2877626796-52576-3234-0000 I1120 15:13:39.929323 1682874368 process_isolator.cpp:100] Launching default (/Users/vinod/workspace/apache/mesos/build/src/examples/java/test-executor) in /tmp/ExamplesTest_JavaFramework_wSc7u8/2/slaves/201311201513-2877626796-52576-3234-2/frameworks/201311201513-2877626796-52576-3234-0000/executors/default/runs/2bd0e75d-a2b9-4ae6-be08-9782612309a5 with resources ' for framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:39.931243 1682874368 process_isolator.cpp:163] Forked executor at 3269 I1120 15:13:39.931612 1681801216 slave.cpp:2073] Monitoring executor default of framework 201311201513-2877626796-52576-3234-0000 forked at pid 3269 E1120 15:13:39.931836 1681801216 slave.cpp:2099] Failed to watch executor default of framework 201311201513-2877626796-52576-3234-0000: Already watched I1120 15:13:39.936460 1680728064 slave.cpp:943] Queuing task '2' for executor default of framework '201311201513-2877626796-52576-3234-0000 I1120 15:13:39.936619 1681801216 process_isolator.cpp:100] Launching default (/Users/vinod/workspace/apache/mesos/build/src/examples/java/test-executor) in /tmp/ExamplesTest_JavaFramework_wSc7u8/0/slaves/201311201513-2877626796-52576-3234-0/frameworks/201311201513-2877626796-52576-3234-0000/executors/default/runs/16d600da-da86-4614-91cb-58a7b27ab534 with resources ' for framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:39.941299 1681801216 process_isolator.cpp:163] Forked executor at 3270 I1120 15:13:39.942179 1681801216 slave.cpp:2073] Monitoring executor default of framework 201311201513-2877626796-52576-3234-0000 forked at pid 3270 E1120 15:13:39.942395 1681801216 slave.cpp:2099] Failed to watch executor default of framework 201311201513-2877626796-52576-3234-0000: Already watched Fetching resources into '/tmp/ExamplesTest_JavaFramework_wSc7u8/2/slaves/201311201513-2877626796-52576-3234-2/frameworks/201311201513-2877626796-52576-3234-0000/executors/default/runs/2bd0e75d-a2b9-4ae6-be08-9782612309a5' Fetching resources into '/tmp/ExamplesTest_JavaFramework_wSc7u8/1/slaves/201311201513-2877626796-52576-3234-1/frameworks/201311201513-2877626796-52576-3234-0000/executors/default/runs/375b31a9-7093-4db1-964d-e6b425b1e4b4' Fetching resources into '/tmp/ExamplesTest_JavaFramework_wSc7u8/0/slaves/201311201513-2877626796-52576-3234-0/frameworks/201311201513-2877626796-52576-3234-0000/executors/default/runs/16d600da-da86-4614-91cb-58a7b27ab534' I1120 15:13:40.372573 1681801216 slave.cpp:1406] Got registration for executor 'default' of framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:40.373258 1681801216 slave.cpp:1527] Flushing queued task 1 for executor 'default' of framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:40.388317 1681801216 slave.cpp:1406] Got registration for executor 'default' of framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:40.388983 1681801216 slave.cpp:1527] Flushing queued task 0 for executor 'default' of framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:40.398084 1679654912 slave.cpp:1406] Got registration for executor 'default' of framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:40.399344 1679654912 slave.cpp:1527] Flushing queued task 2 for executor 'default' of framework 201311201513-2877626796-52576-3234-0000 Registered executor on vkone.local I1120 15:13:40.491843 1679654912 slave.cpp:1740] Handling status update TASK_RUNNING (UUID: f04b1852-3669-444a-906f-3675f784c14f) for task 1 of framework 201311201513-2877626796-52576-3234-0000 from executor(1)@172.25.133.171:52577 I1120 15:13:40.492202 1679654912 status_update_manager.cpp:305] Received status update TASK_RUNNING (UUID: f04b1852-3669-444a-906f-3675f784c14f) for task 1 of framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:40.492424 1679654912 status_update_manager.cpp:356] Forwarding status update TASK_RUNNING (UUID: f04b1852-3669-444a-906f-3675f784c14f) for task 1 of framework 201311201513-2877626796-52576-3234-0000 to master@172.25.133.171:52576 Registered executor on vkone.local I1120 15:13:40.492671 1682337792 master.cpp:1452] Status update TASK_RUNNING (UUID: f04b1852-3669-444a-906f-3675f784c14f) for task 1 of framework 201311201513-2877626796-52576-3234-0000 from slave(3)@172.25.133.171:52576 I1120 15:13:40.492735 1682337792 slave.cpp:1865] Sending acknowledgement for status update TASK_RUNNING (UUID: f04b1852-3669-444a-906f-3675f784c14f) for task 1 of framework 201311201513-2877626796-52576-3234-0000 to executor(1)@172.25.133.171:52577 Status update: task 1 is in state TASK_RUNNING I1120 15:13:40.502235 1679654912 status_update_manager.cpp:380] Received status update acknowledgement (UUID: f04b1852-3669-444a-906f-3675f784c14f) for task 1 of framework 201311201513-2877626796-52576-3234-0000 Registered executor on vkone.local I1120 15:13:40.531292 1679654912 slave.cpp:1740] Handling status update TASK_RUNNING (UUID: c19b6a5a-19ce-4613-8a5a-08fe807ff27c) for task 2 of framework 201311201513-2877626796-52576-3234-0000 from executor(1)@172.25.133.171:52579 I1120 15:13:40.532091 1680728064 status_update_manager.cpp:305] Received status update TASK_RUNNING (UUID: c19b6a5a-19ce-4613-8a5a-08fe807ff27c) for task 2 of framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:40.532305 1680728064 status_update_manager.cpp:356] Forwarding status update TASK_RUNNING (UUID: c19b6a5a-19ce-4613-8a5a-08fe807ff27c) for task 2 of framework 201311201513-2877626796-52576-3234-0000 to master@172.25.133.171:52576 I1120 15:13:40.532776 1682874368 slave.cpp:1865] Sending acknowledgement for status update TASK_RUNNING (UUID: c19b6a5a-19ce-4613-8a5a-08fe807ff27c) for task 2 of framework 201311201513-2877626796-52576-3234-0000 to executor(1)@172.25.133.171:52579 I1120 15:13:40.532951 1681801216 master.cpp:1452] Status update TASK_RUNNING (UUID: c19b6a5a-19ce-4613-8a5a-08fe807ff27c) for task 2 of framework 201311201513-2877626796-52576-3234-0000 from slave(1)@172.25.133.171:52576 Status update: task 2 is in state TASK_RUNNING I1120 15:13:40.538895 1682874368 status_update_manager.cpp:380] Received status update acknowledgement (UUID: c19b6a5a-19ce-4613-8a5a-08fe807ff27c) for task 2 of framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:40.541267 1682874368 slave.cpp:1740] Handling status update TASK_RUNNING (UUID: c218b0c3-d77c-4901-8570-391c330ba117) for task 0 of framework 201311201513-2877626796-52576-3234-0000 from executor(1)@172.25.133.171:52578 I1120 15:13:40.541555 1682874368 status_update_manager.cpp:305] Received status update TASK_RUNNING (UUID: c218b0c3-d77c-4901-8570-391c330ba117) for task 0 of framework 201311201513-2877626796-52576-3234-0000 I1120 15:13:40.541725 1682874368 status_update_manager.cpp:356] Forwarding status update TASK_RUNNING (UUID: c218b0c3-d77c-4901-8570-391c330ba117) for task 0 of framework 201311201513-2877626796-52576-3234-0000 to master@172.25.133.171:52576 I1120 15:13:40.542196 1682874368 master.cpp:1452] Status update TASK_RUNNING (UUID: c218b0c3-d77c-4901-8570-391c330ba117) for task 0 of framework 201311201513-2877626796-52576-3234-0000 from slave(2)@172.25.133.171:52576 I1120 15:13:40.542251 1682874368 slave.cpp:1865] Sending acknowledgement for status update TASK_RUNNING (UUID: c218b0c3-d77c-4901-8570-391c330ba117) for task 0 of framework 201311201513-2877626796-52576-3234-0000 to executor(1)@172.25.133.171:52578 Status update: task 0 is in state TASK_RUNNING I1120 15:13:40.545537 1682874368 status_update_manager.cpp:380] Received status update acknowledgement (UUID: c218b0c3-d77c-4901-8570-391c330ba117) for task 0 of... | 8 |
| 12 | MESOS-920 | Set GLOG_drop_log_memory=false in environment prior to logging initialization. | We've observed issues where the masters are slow to respond. Two perf traces collected while the masters were slow to respond: {noformat} 25.84% [kernel] [k] default_send_IPI_mask_sequence_phys 20.44% [kernel] [k] native_write_msr_safe 4.54% [kernel] [k] _raw_spin_lock 2.95% libc-2.5.so [.] _int_malloc 1.82% libc-2.5.so [.] malloc 1.55% [kernel] [k] apic_timer_interrupt 1.36% libc-2.5.so [.] _int_free {noformat} {noformat} 29.03% [kernel] [k] default_send_IPI_mask_sequence_phys 9.64% [kernel] [k] _raw_spin_lock 7.38% [kernel] [k] native_write_msr_safe 2.43% libc-2.5.so [.] _int_malloc 2.05% libc-2.5.so [.] _int_free 1.67% [kernel] [k] apic_timer_interrupt 1.58% libc-2.5.so [.] malloc {noformat} These have been found to be attributed to the posix_fadvise calls made by glog. We can disable these via the environment: {noformat} GLOG_DEFINE_bool(drop_log_memory, true, "Drop in-memory buffers of log contents. " "Logs can grow very quickly and they are rarely read before they " "need to be evicted from memory. Instead, drop them from memory " "as soon as they are flushed to disk."); {noformat} {code} if (FLAGS_drop_log_memory) { if (file_length_ >= logging::kPageSize) { // don't evict the most recent page uint32 len = file_length_ & ~(logging::kPageSize - 1); posix_fadvise(fileno(file_), 0, len, POSIX_FADV_DONTNEED); } } {code} We should set GLOG_drop_log_memory=false prior to making our call to google::InitGoogleLogging, to avoid others running into this issue. | 2 |
| 13 | MESOS-934 | 'Logging and Debugging' document is out-of-date. | The following is no longer correct: http://mesos.apache.org/documentation/latest/logging-and-debugging/ We should either delete this document or re-write it entirely. | 1 |
| 14 | MESOS-976 | SlaveRecoveryTest/1.SchedulerFailover is flaky | [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from SlaveRecoveryTest/1, where TypeParam = mesos::internal::slave::CgroupsIsolator [ RUN ] SlaveRecoveryTest/1.SchedulerFailover I0206 20:18:31.525116 56447 master.cpp:239] Master ID: 2014-02-06-20:18:31-1740121354-55566-56447 Hostname: smfd-bkq-03-sr4.devel.twitter.com I0206 20:18:31.525295 56481 master.cpp:321] Master started on 10.37.184.103:55566 I0206 20:18:31.525315 56481 master.cpp:324] Master only allowing authenticated frameworks to register! I0206 20:18:31.527093 56481 master.cpp:756] The newly elected leader is master@10.37.184.103:55566 I0206 20:18:31.527122 56481 master.cpp:764] Elected as the leading master! I0206 20:18:31.530642 56473 slave.cpp:112] Slave started on 9)@10.37.184.103:55566 I0206 20:18:31.530802 56473 slave.cpp:212] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0206 20:18:31.531203 56473 slave.cpp:240] Slave hostname: smfd-bkq-03-sr4.devel.twitter.com I0206 20:18:31.531221 56473 slave.cpp:241] Slave checkpoint: true I0206 20:18:31.531991 56482 cgroups_isolator.cpp:225] Using /tmp/mesos_test_cgroup as cgroups hierarchy root I0206 20:18:31.532470 56478 state.cpp:33] Recovering state from '/tmp/SlaveRecoveryTest_1_SchedulerFailover_7dC2N1/meta' I0206 20:18:31.532698 56469 status_update_manager.cpp:188] Recovering status update manager I0206 20:18:31.533962 56472 sched.cpp:265] Authenticating with master master@10.37.184.103:55566 I0206 20:18:31.534102 56472 sched.cpp:234] Detecting new master I0206 20:18:31.534124 56484 authenticatee.hpp:124] Creating new client SASL connection I0206 20:18:31.534299 56473 master.cpp:2317] Authenticating framework at scheduler(9)@10.37.184.103:55566 I0206 20:18:31.534459 56461 authenticator.hpp:140] Creating new server SASL connection I0206 20:18:31.534572 56466 authenticatee.hpp:212] Received SASL authentication mechanisms: CRAM-MD5 I0206 20:18:31.534595 56466 authenticatee.hpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I0206 20:18:31.534667 56474 authenticator.hpp:243] Received SASL authentication start I0206 20:18:31.534732 56474 authenticator.hpp:325] Authentication requires more steps I0206 20:18:31.534814 56468 authenticatee.hpp:258] Received SASL authentication step I0206 20:18:31.534946 56466 authenticator.hpp:271] Received SASL authentication step I0206 20:18:31.535007 56466 authenticator.hpp:317] Authentication success I0206 20:18:31.535084 56471 authenticatee.hpp:298] Authentication success I0206 20:18:31.535107 56461 master.cpp:2357] Successfully authenticated framework at scheduler(9)@10.37.184.103:55566 I0206 20:18:31.535392 56476 sched.cpp:339] Successfully authenticated with master master@10.37.184.103:55566 I0206 20:18:31.535512 56465 master.cpp:812] Received registration request from scheduler(9)@10.37.184.103:55566 I0206 20:18:31.535570 56465 master.cpp:830] Registering framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 at scheduler(9)@10.37.184.103:55566 I0206 20:18:31.535856 56465 hierarchical_allocator_process.hpp:332] Added framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.537802 56482 cgroups_isolator.cpp:840] Recovering isolator I0206 20:18:31.538462 56472 slave.cpp:2760] Finished recovery I0206 20:18:31.538910 56472 slave.cpp:508] New master detected at master@10.37.184.103:55566 I0206 20:18:31.539036 56478 status_update_manager.cpp:162] New master detected at master@10.37.184.103:55566 I0206 20:18:31.539223 56464 master.cpp:1834] Attempting to register slave on smfd-bkq-03-sr4.devel.twitter.com at slave(9)@10.37.184.103:55566 I0206 20:18:31.539271 56472 slave.cpp:533] Detecting new master I0206 20:18:31.539330 56464 master.cpp:2804] Adding slave 2014-02-06-20:18:31-1740121354-55566-56447-0 at smfd-bkq-03-sr4.devel.twitter.com with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0206 20:18:31.539454 56472 slave.cpp:551] Registered with master master@10.37.184.103:55566; given slave ID 2014-02-06-20:18:31-1740121354-55566-56447-0 I0206 20:18:31.539620 56472 slave.cpp:564] Checkpointing SlaveInfo to '/tmp/SlaveRecoveryTest_1_SchedulerFailover_7dC2N1/meta/slaves/2014-02-06-20:18:31-1740121354-55566-56447-0/slave.info' I0206 20:18:31.539834 56475 hierarchical_allocator_process.hpp:445] Added slave 2014-02-06-20:18:31-1740121354-55566-56447-0 (smfd-bkq-03-sr4.devel.twitter.com) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] available) I0206 20:18:31.540341 56472 master.cpp:2272] Sending 1 offers to framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.543433 56472 master.cpp:1568] Processing reply for offers: [ 2014-02-06-20:18:31-1740121354-55566-56447-0 ] on slave 2014-02-06-20:18:31-1740121354-55566-56447-0 (smfd-bkq-03-sr4.devel.twitter.com) for framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.543642 56472 master.hpp:411] Adding task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 2014-02-06-20:18:31-1740121354-55566-56447-0 (smfd-bkq-03-sr4.devel.twitter.com) I0206 20:18:31.543781 56472 master.cpp:2441] Launching task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 2014-02-06-20:18:31-1740121354-55566-56447-0 (smfd-bkq-03-sr4.devel.twitter.com) I0206 20:18:31.544002 56484 slave.cpp:736] Got assigned task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 for framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.544097 56484 slave.cpp:2899] Checkpointing FrameworkInfo to '/tmp/SlaveRecoveryTest_1_SchedulerFailover_7dC2N1/meta/slaves/2014-02-06-20:18:31-1740121354-55566-56447-0/frameworks/2014-02-06-20:18:31-1740121354-55566-56447-0000/framework.info' I0206 20:18:31.544272 56484 slave.cpp:2906] Checkpointing framework pid 'scheduler(9)@10.37.184.103:55566' to '/tmp/SlaveRecoveryTest_1_SchedulerFailover_7dC2N1/meta/slaves/2014-02-06-20:18:31-1740121354-55566-56447-0/frameworks/2014-02-06-20:18:31-1740121354-55566-56447-0000/framework.pid' I0206 20:18:31.544617 56484 slave.cpp:845] Launching task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 for framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.546721 56484 slave.cpp:3169] Checkpointing ExecutorInfo to '/tmp/SlaveRecoveryTest_1_SchedulerFailover_7dC2N1/meta/slaves/2014-02-06-20:18:31-1740121354-55566-56447-0/frameworks/2014-02-06-20:18:31-1740121354-55566-56447-0000/executors/d045a0bd-2ed2-410a-bd1f-5bd9219896e3/executor.info' I0206 20:18:31.547317 56484 slave.cpp:3257] Checkpointing TaskInfo to '/tmp/SlaveRecoveryTest_1_SchedulerFailover_7dC2N1/meta/slaves/2014-02-06-20:18:31-1740121354-55566-56447-0/frameworks/2014-02-06-20:18:31-1740121354-55566-56447-0000/executors/d045a0bd-2ed2-410a-bd1f-5bd9219896e3/runs/9adabe16-5d84-45c9-bc83-1a72a6d1c986/tasks/d045a0bd-2ed2-410a-bd1f-5bd9219896e3/task.info' I0206 20:18:31.547514 56484 slave.cpp:955] Queuing task 'd045a0bd-2ed2-410a-bd1f-5bd9219896e3' for executor d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework '2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.547590 56481 cgroups_isolator.cpp:517] Launching d045a0bd-2ed2-410a-bd1f-5bd9219896e3 (/home/vinod/mesos/build/src/mesos-executor) in /tmp/SlaveRecoveryTest_1_SchedulerFailover_7dC2N1/slaves/2014-02-06-20:18:31-1740121354-55566-56447-0/frameworks/2014-02-06-20:18:31-1740121354-55566-56447-0000/executors/d045a0bd-2ed2-410a-bd1f-5bd9219896e3/runs/9adabe16-5d84-45c9-bc83-1a72a6d1c986 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] for framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 in cgroup mesos_test/framework_2014-02-06-20:18:31-1740121354-55566-56447-0000_executor_d045a0bd-2ed2-410a-bd1f-5bd9219896e3_tag_9adabe16-5d84-45c9-bc83-1a72a6d1c986 I0206 20:18:31.548408 56481 cgroups_isolator.cpp:717] Changing cgroup controls for executor d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0206 20:18:31.548833 56481 cgroups_isolator.cpp:1007] Updated 'cpu.shares' to 2048 for executor d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.549294 56481 cgroups_isolator.cpp:1117] Updated 'memory.soft_limit_in_bytes' to 1GB for executor d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.550107 56481 cgroups_isolator.cpp:1147] Updated 'memory.limit_in_bytes' to 1GB for executor d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.550571 56481 cgroups_isolator.cpp:1174] Started listening for OOM events for executor d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.551553 56481 cgroups_isolator.cpp:569] Forked executor at = 56671 Checkpointing executor's forked pid 56671 to '/tmp/SlaveRecoveryTest_1_SchedulerFailover_7dC2N1/meta/slaves/2014-02-06-20:18:31-1740121354-55566-56447-0/frameworks/2014-02-06-20:18:31-1740121354-55566-56447-0000/executors/d045a0bd-2ed2-410a-bd1f-5bd9219896e3/runs/9adabe16-5d84-45c9-bc83-1a72a6d1c986/pids/forked.pid' I0206 20:18:31.552222 56472 slave.cpp:2098] Monitoring executor d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 forked at pid 56671 Fetching resources into '/tmp/SlaveRecoveryTest_1_SchedulerFailover_7dC2N1/slaves/2014-02-06-20:18:31-1740121354-55566-56447-0/frameworks/2014-02-06-20:18:31-1740121354-55566-56447-0000/executors/d045a0bd-2ed2-410a-bd1f-5bd9219896e3/runs/9adabe16-5d84-45c9-bc83-1a72a6d1c986' I0206 20:18:31.604012 56472 slave.cpp:1431] Got registration for executor 'd045a0bd-2ed2-410a-bd1f-5bd9219896e3' of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.604167 56472 slave.cpp:1516] Checkpointing executor pid 'executor(1)@10.37.184.103:46181' to '/tmp/SlaveRecoveryTest_1_SchedulerFailover_7dC2N1/meta/slaves/2014-02-06-20:18:31-1740121354-55566-56447-0/frameworks/2014-02-06-20:18:31-1740121354-55566-56447-0000/executors/d045a0bd-2ed2-410a-bd1f-5bd9219896e3/runs/9adabe16-5d84-45c9-bc83-1a72a6d1c986/pids/libprocess.pid' I0206 20:18:31.605183 56472 slave.cpp:1552] Flushing queued task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 for executor 'd045a0bd-2ed2-410a-bd1f-5bd9219896e3' of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 Registered executor on smfd-bkq-03-sr4.devel.twitter.com Starting task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 sh -c 'sleep 1000' Forked command at 56712 I0206 20:18:31.613098 56481 slave.cpp:1765] Handling status update TASK_RUNNING (UUID: fc151a46-751b-4c4b-b048-1727752f34e3) for task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 from executor(1)@10.37.184.103:46181 I0206 20:18:31.613628 56469 status_update_manager.cpp:314] Received status update TASK_RUNNING (UUID: fc151a46-751b-4c4b-b048-1727752f34e3) for task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.614006 56469 status_update_manager.hpp:342] Checkpointing UPDATE for status update TASK_RUNNING (UUID: fc151a46-751b-4c4b-b048-1727752f34e3) for task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.795529 56469 status_update_manager.cpp:367] Forwarding status update TASK_RUNNING (UUID: fc151a46-751b-4c4b-b048-1727752f34e3) for task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 to master@10.37.184.103:55566 I0206 20:18:31.795992 56480 slave.cpp:1890] Sending acknowledgement for status update TASK_RUNNING (UUID: fc151a46-751b-4c4b-b048-1727752f34e3) for task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 to executor(1)@10.37.184.103:46181 I0206 20:18:31.796131 56471 master.cpp:2020] Status update TASK_RUNNING (UUID: fc151a46-751b-4c4b-b048-1727752f34e3) for task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 from slave(9)@10.37.184.103:55566 I0206 20:18:31.797099 56483 status_update_manager.cpp:392] Received status update acknowledgement (UUID: fc151a46-751b-4c4b-b048-1727752f34e3) for task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.797165 56483 status_update_manager.hpp:342] Checkpointing ACK for status update TASK_RUNNING (UUID: fc151a46-751b-4c4b-b048-1727752f34e3) for task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.882767 56481 slave.cpp:394] Slave terminating I0206 20:18:31.883112 56481 master.cpp:641] Slave 2014-02-06-20:18:31-1740121354-55566-56447-0 (smfd-bkq-03-sr4.devel.twitter.com) disconnected I0206 20:18:31.883200 56476 hierarchical_allocator_process.hpp:484] Slave 2014-02-06-20:18:31-1740121354-55566-56447-0 disconnected I0206 20:18:31.888206 56473 sched.cpp:265] Authenticating with master master@10.37.184.103:55566 I0206 20:18:31.888473 56473 sched.cpp:234] Detecting new master I0206 20:18:31.888556 56469 authenticatee.hpp:124] Creating new client SASL connection I0206 20:18:31.888978 56484 master.cpp:2317] Authenticating framework at scheduler(10)@10.37.184.103:55566 I0206 20:18:31.889348 56469 authenticator.hpp:140] Creating new server SASL connection I0206 20:18:31.889925 56469 authenticatee.hpp:212] Received SASL authentication mechanisms: CRAM-MD5 I0206 20:18:31.889989 56469 authenticatee.hpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I0206 20:18:31.890059 56469 authenticator.hpp:243] Received SASL authentication start I0206 20:18:31.890233 56469 authenticator.hpp:325] Authentication requires more steps I0206 20:18:31.890399 56468 authenticatee.hpp:258] Received SASL authentication step I0206 20:18:31.890554 56484 authenticator.hpp:271] Received SASL authentication step I0206 20:18:31.890630 56484 authenticator.hpp:317] Authentication success I0206 20:18:31.890728 56470 authenticatee.hpp:298] Authentication success I0206 20:18:31.890748 56484 master.cpp:2357] Successfully authenticated framework at scheduler(10)@10.37.184.103:55566 I0206 20:18:31.892210 56469 sched.cpp:339] Successfully authenticated with master master@10.37.184.103:55566 I0206 20:18:31.892410 56473 master.cpp:900] Re-registering framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 at scheduler(10)@10.37.184.103:55566 I0206 20:18:31.892460 56473 master.cpp:926] Framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 failed over W0206 20:18:31.892691 56465 master.cpp:1048] Ignoring deactivate framework message for framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 from 'scheduler(9)@10.37.184.103:55566' because it is not from the registered framework 'scheduler(10)@10.37.184.103:55566' I0206 20:18:31.897049 56466 slave.cpp:112] Slave started on 10)@10.37.184.103:55566 I0206 20:18:31.897207 56466 slave.cpp:212] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0206 20:18:31.897536 56466 slave.cpp:240] Slave hostname: smfd-bkq-03-sr4.devel.twitter.com I0206 20:18:31.897554 56466 slave.cpp:241] Slave checkpoint: true I0206 20:18:31.898388 56463 cgroups_isolator.cpp:225] Using /tmp/mesos_test_cgroup as cgroups hierarchy root I0206 20:18:31.898936 56472 state.cpp:33] Recovering state from '/tmp/SlaveRecoveryTest_1_SchedulerFailover_7dC2N1/meta' I0206 20:18:31.901702 56465 slave.cpp:2828] Recovering framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.901759 56465 slave.cpp:3020] Recovering executor 'd045a0bd-2ed2-410a-bd1f-5bd9219896e3' of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:31.902716 56464 status_update_manager.cpp:188] Recovering status update manager I0206 20:18:31.902884 56464 status_update_manager.cpp:196] Recovering executor 'd045a0bd-2ed2-410a-bd1f-5bd9219896e3' of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:34.475915 56463 cgroups_isolator.cpp:840] Recovering isolator I0206 20:18:34.476066 56463 cgroups_isolator.cpp:847] Recovering executor 'd045a0bd-2ed2-410a-bd1f-5bd9219896e3' of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:34.477478 56463 cgroups_isolator.cpp:1174] Started listening for OOM events for executor d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:34.478728 56463 slave.cpp:2700] Sending reconnect request to executor d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 at executor(1)@10.37.184.103:46181 I0206 20:18:34.480114 56476 slave.cpp:1597] Re-registering executor d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:34.480566 56476 cgroups_isolator.cpp:717] Changing cgroup controls for executor d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0206 20:18:34.481370 56476 cgroups_isolator.cpp:1007] Updated 'cpu.shares' to 2048 for executor d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:34.481827 56476 cgroups_isolator.cpp:1117] Updated 'memory.soft_limit_in_bytes' to 1GB for executor d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 Re-registered executor on smfd-bkq-03-sr4.devel.twitter.com I0206 20:18:34.489497 56471 slave.cpp:1713] Cleaning up un-reregistered executors I0206 20:18:34.489588 56471 slave.cpp:2760] Finished recovery I0206 20:18:34.490048 56463 slave.cpp:508] New master detected at master@10.37.184.103:55566 I0206 20:18:34.490257 56475 status_update_manager.cpp:162] New master detected at master@10.37.184.103:55566 I0206 20:18:34.490357 56463 slave.cpp:533] Detecting new master W0206 20:18:34.490603 56480 master.cpp:1878] Slave at slave(10)@10.37.184.103:55566 (smfd-bkq-03-sr4.devel.twitter.com) is being allowed to re-register with an already in use id (2014-02-06-20:18:31-1740121354-55566-56447-0) I0206 20:18:34.490927 56479 slave.cpp:601] Re-registered with master master@10.37.184.103:55566 I0206 20:18:34.491322 56461 hierarchical_allocator_process.hpp:498] Slave 2014-02-06-20:18:31-1740121354-55566-56447-0 reconnected I0206 20:18:34.491421 56468 slave.cpp:1312] Updating framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 pid to scheduler(10)@10.37.184.103:55566 I0206 20:18:34.491444 56480 master.cpp:1673] Asked to kill task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:34.491488 56468 slave.cpp:1320] Checkpointing framework pid 'scheduler(10)@10.37.184.103:55566' to '/tmp/SlaveRecoveryTest_1_SchedulerFailover_7dC2N1/meta/slaves/2014-02-06-20:18:31-1740121354-55566-56447-0/frameworks/2014-02-06-20:18:31-1740121354-55566-56447-0000/framework.pid' I0206 20:18:34.491497 56480 master.cpp:1707] Telling slave 2014-02-06-20:18:31-1740121354-55566-56447-0 (smfd-bkq-03-sr4.devel.twitter.com) to kill task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 I0206 20:18:34.491657 56468 slave.cpp:1013] Asked to kill task d045a0bd-2ed2-410a-bd1f-5bd9219896e3 of framework 2014-02-06-20:18:31-1740121354-55566-56447-0000 Shutting down Killing process tree at pid 56712... | 1 |
| 15 | MESOS-988 | ExamplesTest.PythonFramework is flaky | Looks like a SEGFAULT during shutdown. {noformat} [ RUN ] ExamplesTest.PythonFramework Using temporary directory '/tmp/ExamplesTest_PythonFramework_RZ4yaf' WARNING: Logging before InitGoogleLogging() is written to STDERR I0211 21:14:47.861803 21045 process.cpp:1591] libprocess is initialized on 67.195.138.9:53443 for 8 cpus I0211 21:14:47.861884 21045 logging.cpp:140] Logging to STDERR I0211 21:14:47.862761 21045 master.cpp:240] Master ID: 2014-02-11-21:14:47-160088899-53443-21045 Hostname: vesta.apache.org I0211 21:14:47.862897 21054 master.cpp:322] Master started on 67.195.138.9:53443 I0211 21:14:47.862908 21054 master.cpp:325] Master only allowing authenticated frameworks to register! I0211 21:14:47.864362 21053 master.cpp:86] No whitelist given. Advertising offers for all slaves I0211 21:14:47.864506 21055 slave.cpp:112] Slave started on 1)@67.195.138.9:53443 I0211 21:14:47.864522 21059 slave.cpp:112] Slave started on 2)@67.195.138.9:53443 I0211 21:14:47.864749 21055 slave.cpp:212] Slave resources: cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] I0211 21:14:47.864778 21059 slave.cpp:212] Slave resources: cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] I0211 21:14:47.864819 21055 slave.cpp:240] Slave hostname: vesta.apache.org I0211 21:14:47.864827 21055 slave.cpp:241] Slave checkpoint: true I0211 21:14:47.864850 21059 slave.cpp:240] Slave hostname: vesta.apache.org I0211 21:14:47.864858 21059 slave.cpp:241] Slave checkpoint: true I0211 21:14:47.865329 21055 master.cpp:760] The newly elected leader is master@67.195.138.9:53443 with id 2014-02-11-21:14:47-160088899-53443-21045 I0211 21:14:47.865350 21055 master.cpp:770] Elected as the leading master! I0211 21:14:47.865399 21055 state.cpp:33] Recovering state from '/tmp/mesos-Z8v6cu/1/meta' I0211 21:14:47.865407 21059 state.cpp:33] Recovering state from '/tmp/mesos-Z8v6cu/0/meta' I0211 21:14:47.865502 21052 hierarchical_allocator_process.hpp:302] Initializing hierarchical allocator process with master : master@67.195.138.9:53443 I0211 21:14:47.865540 21054 status_update_manager.cpp:188] Recovering status update manager I0211 21:14:47.865619 21053 process_isolator.cpp:319] Recovering isolator I0211 21:14:47.865674 21057 status_update_manager.cpp:188] Recovering status update manager I0211 21:14:47.865699 21059 slave.cpp:2760] Finished recovery I0211 21:14:47.865733 21053 process_isolator.cpp:319] Recovering isolator I0211 21:14:47.865789 21053 slave.cpp:2760] Finished recovery I0211 21:14:47.865921 21059 slave.cpp:508] New master detected at master@67.195.138.9:53443 I0211 21:14:47.865958 21053 status_update_manager.cpp:162] New master detected at master@67.195.138.9:53443 I0211 21:14:47.865978 21059 slave.cpp:533] Detecting new master I0211 21:14:47.866019 21053 slave.cpp:508] New master detected at master@67.195.138.9:53443 I0211 21:14:47.866063 21053 slave.cpp:533] Detecting new master I0211 21:14:47.866070 21055 status_update_manager.cpp:162] New master detected at master@67.195.138.9:53443 I0211 21:14:47.866077 21059 master.cpp:1840] Attempting to register slave on vesta.apache.org at slave(2)@67.195.138.9:53443 I0211 21:14:47.866092 21059 master.cpp:2810] Adding slave 2014-02-11-21:14:47-160088899-53443-21045-0 at vesta.apache.org with cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] I0211 21:14:47.866216 21059 master.cpp:1840] Attempting to register slave on vesta.apache.org at slave(1)@67.195.138.9:53443 I0211 21:14:47.866225 21053 slave.cpp:551] Registered with master master@67.195.138.9:53443; given slave ID 2014-02-11-21:14:47-160088899-53443-21045-0 I0211 21:14:47.866228 21059 master.cpp:2810] Adding slave 2014-02-11-21:14:47-160088899-53443-21045-1 at vesta.apache.org with cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] I0211 21:14:47.866278 21055 hierarchical_allocator_process.hpp:445] Added slave 2014-02-11-21:14:47-160088899-53443-21045-0 (vesta.apache.org) with cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] (and cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] available) I0211 21:14:47.866297 21059 slave.cpp:551] Registered with master master@67.195.138.9:53443; given slave ID 2014-02-11-21:14:47-160088899-53443-21045-1 I0211 21:14:47.866327 21055 hierarchical_allocator_process.hpp:708] Performed allocation for slave 2014-02-11-21:14:47-160088899-53443-21045-0 in 11us I0211 21:14:47.866330 21053 slave.cpp:564] Checkpointing SlaveInfo to '/tmp/mesos-Z8v6cu/1/meta/slaves/2014-02-11-21:14:47-160088899-53443-21045-0/slave.info' I0211 21:14:47.866400 21059 slave.cpp:564] Checkpointing SlaveInfo to '/tmp/mesos-Z8v6cu/0/meta/slaves/2014-02-11-21:14:47-160088899-53443-21045-1/slave.info' I0211 21:14:47.866399 21055 hierarchical_allocator_process.hpp:445] Added slave 2014-02-11-21:14:47-160088899-53443-21045-1 (vesta.apache.org) with cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] (and cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] available) I0211 21:14:47.866423 21055 hierarchical_allocator_process.hpp:708] Performed allocation for slave 2014-02-11-21:14:47-160088899-53443-21045-1 in 2505ns I0211 21:14:47.866636 21059 slave.cpp:112] Slave started on 3)@67.195.138.9:53443 I0211 21:14:47.866727 21059 slave.cpp:212] Slave resources: cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] I0211 21:14:47.866766 21059 slave.cpp:240] Slave hostname: vesta.apache.org I0211 21:14:47.866772 21059 slave.cpp:241] Slave checkpoint: true I0211 21:14:47.867300 21052 state.cpp:33] Recovering state from '/tmp/mesos-Z8v6cu/2/meta' I0211 21:14:47.867368 21052 status_update_manager.cpp:188] Recovering status update manager I0211 21:14:47.867419 21055 process_isolator.cpp:319] Recovering isolator I0211 21:14:47.867544 21052 slave.cpp:2760] Finished recovery I0211 21:14:47.867729 21052 slave.cpp:508] New master detected at master@67.195.138.9:53443 I0211 21:14:47.867770 21054 status_update_manager.cpp:162] New master detected at master@67.195.138.9:53443 I0211 21:14:47.867777 21052 slave.cpp:533] Detecting new master I0211 21:14:47.867815 21055 master.cpp:1840] Attempting to register slave on vesta.apache.org at slave(3)@67.195.138.9:53443 I0211 21:14:47.867827 21055 master.cpp:2810] Adding slave 2014-02-11-21:14:47-160088899-53443-21045-2 at vesta.apache.org with cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] I0211 21:14:47.867885 21052 slave.cpp:551] Registered with master master@67.195.138.9:53443; given slave ID 2014-02-11-21:14:47-160088899-53443-21045-2 I0211 21:14:47.867961 21055 hierarchical_allocator_process.hpp:445] Added slave 2014-02-11-21:14:47-160088899-53443-21045-2 (vesta.apache.org) with cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] (and cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] available) I0211 21:14:47.867985 21052 slave.cpp:564] Checkpointing SlaveInfo to '/tmp/mesos-Z8v6cu/2/meta/slaves/2014-02-11-21:14:47-160088899-53443-21045-2/slave.info' I0211 21:14:47.867987 21055 hierarchical_allocator_process.hpp:708] Performed allocation for slave 2014-02-11-21:14:47-160088899-53443-21045-2 in 3308ns I0211 21:14:47.868468 21045 sched.cpp:121] Version: 0.18.0 I0211 21:14:47.868633 21055 sched.cpp:217] New master detected at master@67.195.138.9:53443 I0211 21:14:47.868651 21055 sched.cpp:268] Authenticating with master master@67.195.138.9:53443 I0211 21:14:47.868696 21055 sched.cpp:237] Detecting new master I0211 21:14:47.868708 21054 authenticatee.hpp:100] Initializing client SASL I0211 21:14:47.869549 21054 authenticatee.hpp:124] Creating new client SASL connection I0211 21:14:47.869633 21055 master.cpp:2323] Authenticating framework at scheduler(1)@67.195.138.9:53443 I0211 21:14:47.869818 21059 authenticator.hpp:83] Initializing server SASL I0211 21:14:47.870029 21059 auxprop.cpp:45] Initialized in-memory auxiliary property plugin I0211 21:14:47.870040 21059 authenticator.hpp:140] Creating new server SASL connection I0211 21:14:47.870144 21057 authenticatee.hpp:212] Received SASL authentication mechanisms: CRAM-MD5 I0211 21:14:47.870174 21057 authenticatee.hpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I0211 21:14:47.870203 21057 authenticator.hpp:243] Received SASL authentication start I0211 21:14:47.870256 21057 authenticator.hpp:325] Authentication requires more steps I0211 21:14:47.870282 21057 authenticatee.hpp:258] Received SASL authentication step I0211 21:14:47.870348 21057 authenticator.hpp:271] Received SASL authentication step I0211 21:14:47.870376 21057 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'vesta.apache.org' server FQDN: 'vesta.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0211 21:14:47.870384 21057 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0211 21:14:47.870396 21057 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0211 21:14:47.870405 21057 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'vesta.apache.org' server FQDN: 'vesta.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0211 21:14:47.870411 21057 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0211 21:14:47.870415 21057 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0211 21:14:47.870425 21057 authenticator.hpp:317] Authentication success I0211 21:14:47.870445 21057 master.cpp:2363] Successfully authenticated framework at scheduler(1)@67.195.138.9:53443 I0211 21:14:47.870448 21055 authenticatee.hpp:298] Authentication success I0211 21:14:47.870492 21055 sched.cpp:342] Successfully authenticated with master master@67.195.138.9:53443 I0211 21:14:47.870538 21057 master.cpp:818] Received registration request from scheduler(1)@67.195.138.9:53443 I0211 21:14:47.870590 21057 master.cpp:836] Registering framework 2014-02-11-21:14:47-160088899-53443-21045-0000 at scheduler(1)@67.195.138.9:53443 I0211 21:14:47.870661 21055 sched.cpp:391] Framework registered with 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.870661 21057 hierarchical_allocator_process.hpp:332] Added framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.870707 21057 hierarchical_allocator_process.hpp:752] Offering cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] on slave 2014-02-11-21:14:47-160088899-53443-21045-0 to framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.870798 21057 hierarchical_allocator_process.hpp:752] Offering cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] on slave 2014-02-11-21:14:47-160088899-53443-21045-1 to framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.870869 21057 hierarchical_allocator_process.hpp:752] Offering cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] on slave 2014-02-11-21:14:47-160088899-53443-21045-2 to framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.870894 21055 sched.cpp:405] Scheduler::registered took 222149ns I0211 21:14:47.871038 21057 hierarchical_allocator_process.hpp:688] Performed allocation for 3 slaves in 351098ns I0211 21:14:47.871106 21058 master.hpp:439] Adding offer 2014-02-11-21:14:47-160088899-53443-21045-0 with resources cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] on slave 2014-02-11-21:14:47-160088899-53443-21045-2 (vesta.apache.org) I0211 21:14:47.871215 21058 master.hpp:439] Adding offer 2014-02-11-21:14:47-160088899-53443-21045-1 with resources cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] on slave 2014-02-11-21:14:47-160088899-53443-21045-1 (vesta.apache.org) I0211 21:14:47.871296 21058 master.hpp:439] Adding offer 2014-02-11-21:14:47-160088899-53443-21045-2 with resources cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] on slave 2014-02-11-21:14:47-160088899-53443-21045-0 (vesta.apache.org) I0211 21:14:47.871333 21058 master.cpp:2278] Sending 3 offers to framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.873667 21055 sched.cpp:525] Scheduler::resourceOffers took 2.150843ms I0211 21:14:47.873884 21053 master.hpp:449] Removing offer 2014-02-11-21:14:47-160088899-53443-21045-0 with resources cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] on slave 2014-02-11-21:14:47-160088899-53443-21045-2 (vesta.apache.org) I0211 21:14:47.873934 21053 master.cpp:1574] Processing reply for offers: [ 2014-02-11-21:14:47-160088899-53443-21045-0 ] on slave 2014-02-11-21:14:47-160088899-53443-21045-2 (vesta.apache.org) for framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.874035 21053 master.hpp:411] Adding task 0 with resources cpus(*):1; mem(*):32 on slave 2014-02-11-21:14:47-160088899-53443-21045-2 (vesta.apache.org) I0211 21:14:47.874059 21053 master.cpp:2447] Launching task 0 of framework 2014-02-11-21:14:47-160088899-53443-21045-0000 with resources cpus(*):1; mem(*):32 on slave 2014-02-11-21:14:47-160088899-53443-21045-2 (vesta.apache.org) I0211 21:14:47.874150 21059 slave.cpp:736] Got assigned task 0 for framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.874200 21058 hierarchical_allocator_process.hpp:547] Framework 2014-02-11-21:14:47-160088899-53443-21045-0000 left cpus(*):7; mem(*):6929; disk(*):1.38501e+06; ports(*):[31000-32000] unused on slave 2014-02-11-21:14:47-160088899-53443-21045-2 I0211 21:14:47.874250 21053 master.hpp:449] Removing offer 2014-02-11-21:14:47-160088899-53443-21045-1 with resources cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] on slave 2014-02-11-21:14:47-160088899-53443-21045-1 (vesta.apache.org) I0211 21:14:47.874307 21053 master.cpp:1574] Processing reply for offers: [ 2014-02-11-21:14:47-160088899-53443-21045-1 ] on slave 2014-02-11-21:14:47-160088899-53443-21045-1 (vesta.apache.org) for framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.874322 21058 hierarchical_allocator_process.hpp:590] Framework 2014-02-11-21:14:47-160088899-53443-21045-0000 filtered slave 2014-02-11-21:14:47-160088899-53443-21045-2 for 5secs I0211 21:14:47.874354 21059 slave.cpp:845] Launching task 0 for framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.874404 21053 master.hpp:411] Adding task 1 with resources cpus(*):1; mem(*):32 on slave 2014-02-11-21:14:47-160088899-53443-21045-1 (vesta.apache.org) I0211 21:14:47.874428 21053 master.cpp:2447] Launching task 1 of framework 2014-02-11-21:14:47-160088899-53443-21045-0000 with resources cpus(*):1; mem(*):32 on slave 2014-02-11-21:14:47-160088899-53443-21045-1 (vesta.apache.org) I0211 21:14:47.874479 21058 slave.cpp:736] Got assigned task 1 for framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.874586 21053 master.hpp:449] Removing offer 2014-02-11-21:14:47-160088899-53443-21045-2 with resources cpus(*):8; mem(*):6961; disk(*):1.38501e+06; ports(*):[31000-32000] on slave 2014-02-11-21:14:47-160088899-53443-21045-0 (vesta.apache.org) I0211 21:14:47.874646 21053 master.cpp:1574] Processing reply for offers: [ 2014-02-11-21:14:47-160088899-53443-21045-2 ] on slave 2014-02-11-21:14:47-160088899-53443-21045-0 (vesta.apache.org) for framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.874690 21058 slave.cpp:845] Launching task 1 for framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.874694 21053 master.hpp:411] Adding task 2 with resources cpus(*):1; mem(*):32 on slave 2014-02-11-21:14:47-160088899-53443-21045-0 (vesta.apache.org) I0211 21:14:47.874716 21053 master.cpp:2447] Launching task 2 of framework 2014-02-11-21:14:47-160088899-53443-21045-0000 with resources cpus(*):1; mem(*):32 on slave 2014-02-11-21:14:47-160088899-53443-21045-0 (vesta.apache.org) I0211 21:14:47.874820 21053 hierarchical_allocator_process.hpp:547] Framework 2014-02-11-21:14:47-160088899-53443-21045-0000 left cpus(*):7; mem(*):6929; disk(*):1.38501e+06; ports(*):[31000-32000] unused on slave 2014-02-11-21:14:47-160088899-53443-21045-1 I0211 21:14:47.874892 21053 hierarchical_allocator_process.hpp:590] Framework 2014-02-11-21:14:47-160088899-53443-21045-0000 filtered slave 2014-02-11-21:14:47-160088899-53443-21045-1 for 5secs I0211 21:14:47.874922 21053 hierarchical_allocator_process.hpp:547] Framework 2014-02-11-21:14:47-160088899-53443-21045-0000 left cpus(*):7; mem(*):6929; disk(*):1.38501e+06; ports(*):[31000-32000] unused on slave 2014-02-11-21:14:47-160088899-53443-21045-0 I0211 21:14:47.874980 21053 hierarchical_allocator_process.hpp:590] Framework 2014-02-11-21:14:47-160088899-53443-21045-0000 filtered slave 2014-02-11-21:14:47-160088899-53443-21045-0 for 5secs I0211 21:14:47.875012 21053 slave.cpp:736] Got assigned task 2 for framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.875151 21053 slave.cpp:845] Launching task 2 for framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.875527 21059 slave.cpp:955] Queuing task '0' for executor default of framework '2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.875608 21059 process_isolator.cpp:102] Launching default (/home/hudson/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-In-Src-Set-JAVA_HOME/src/examples/python/test-executor) in /tmp/mesos-Z8v6cu/2/slaves/2014-02-11-21:14:47-160088899-53443-21045-2/frameworks/2014-02-11-21:14:47-160088899-53443-21045-0000/executors/default/runs/02cdf8bd-0757-4a40-8e77-af60bb202d71 with resources cpus(*):1; mem(*):32' for framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.876787 21054 slave.cpp:469] Successfully attached file '/tmp/mesos-Z8v6cu/2/slaves/2014-02-11-21:14:47-160088899-53443-21045-2/frameworks/2014-02-11-21:14:47-160088899-53443-21045-0000/executors/default/runs/02cdf8bd-0757-4a40-8e77-af60bb202d71' I0211 21:14:47.876852 21059 process_isolator.cpp:165] Forked executor at 21061 I0211 21:14:47.876940 21058 slave.cpp:955] Queuing task '1' for executor default of framework '2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.877095 21057 process_isolator.cpp:102] Launching default (/home/hudson/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-In-Src-Set-JAVA_HOME/src/examples/python/test-executor) in /tmp/mesos-Z8v6cu/0/slaves/2014-02-11-21:14:47-160088899-53443-21045-1/frameworks/2014-02-11-21:14:47-160088899-53443-21045-0000/executors/default/runs/568b657d-839d-483f-aff1-4872fbfc27dc with resources cpus(*):1; mem(*):32' for framework 2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.877102 21052 slave.cpp:469] Successfully attached file '/tmp/mesos-Z8v6cu/0/slaves/2014-02-11-21:14:47-160088899-53443-21045-1/frameworks/2014-02-11-21:14:47-160088899-53443-21045-0000/executors/default/runs/568b657d-839d-483f-aff1-4872fbfc27dc' I0211 21:14:47.878783 21057 process_isolator.cpp:165] Forked executor at 21062 I0211 21:14:47.879032 21053 slave.cpp:955] Queuing task '2' for executor default of framework '2014-02-11-21:14:47-160088899-53443-21045-0000 I0211 21:14:47.879192 21054 slave.cpp:2098] Monitoring executor default of framework 2014-02-11-21:14:47-160088899-53443-21045-0000 forked at pid 21062 I0211 21:14:47.879192 21058 slave.cpp:469] Successfully attached file '/tmp/mesos-Z8v6cu/1/slaves/2014-02-11-21:14:47-160088899-53443-21045-0/frameworks/2014-02-11-21:14:47-160088899-53443-21045-0000/executors/default/runs/a7c4170a-f40b-4493-81b3-0ea8c70e3977' I0211 21:14:47.879166 21052 process_isolator.cpp:102] Launching default (/home/hudson/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-In-Src-Set-JAVA_HOME/src/examples/p... | 3 |
| 16 | MESOS-998 | Slave should wait until Containerizer::update() completes successfully | Container resources are updated in several places in the slave and we don't check the update was successful or even wait until it completes. | 5 |
| 17 | MESOS-999 | Slave should wait() and start executor registration timeout after launch | The current code will start launch a container and wait on it before the launch is complete. We should do this only after the container has successfully launched. Likewise for the executor registration timeout. | 3 |
| 18 | MESOS-1010 | Python extension build is broken if gflags-dev is installed | In my environment mesos build from master results in broken python api module {{_mesos.so}}: {noformat} nekto0n@ya-darkstar ~/workspace/mesos/src/python $ PYTHONPATH=build/lib.linux-x86_64-2.7/ python -c "import _mesos" Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: /home/nekto0n/workspace/mesos/src/python/build/lib.linux-x86_64-2.7/_mesos.so: undefined symbol: _ZN6google14FlagRegistererC1EPKcS2_S2_S2_PvS3_ {noformat} Unmangled version of symbol looks like this: {noformat} google::FlagRegisterer::FlagRegisterer(char const*, char const*, char const*, char const*, void*, void*) {noformat} During {{./configure}} step {{glog}} finds {{gflags}} development files and starts using them, thus *implicitly* adding dependency on {{libgflags.so}}. This breaks Python extensions module and perhaps can break other mesos subsystems when moved to hosts without {{gflags}} installed. This task is done when the ExamplesTest.PythonFramework test will pass on a system with gflags installed. | 3 |
| 19 | MESOS-1013 | ExamplesTest.JavaLog is flaky | The {{ExamplesTest.JavaLog}} test framework is flaky, possibly related to a race condition between mutexes. {noformat} [ RUN ] ExamplesTest.JavaLog Using temporary directory '/tmp/ExamplesTest_JavaLog_WBWEb9' Feb 18, 2014 12:10:57 PM TestLog main INFO: Starting a local ZooKeeper server ... F0218 12:10:58.575036 17450 coordinator.cpp:394] Check failed: !missing Not expecting local replica to be missing position 3 after the writing is done *** Check failure stack trace: *** tests/script.cpp:81: Failure Failed java_log_test.sh terminated with signal 'Aborted' [ FAILED ] ExamplesTest.JavaLog (2166 ms) {noformat} Full logs attached. | 2 |
| 20 | MESOS-1081 | Master should not deactivate authenticated framework/slave on new AuthenticateMessage unless new authentication succeeds. | Master should not deactivate an authenticated framework/slave upon receiving a new AuthenticateMessage unless new authentication succeeds. As it stands now, a malicious user could spoof the pid of an authenticated framework/slave and send an AuthenticateMessage to knock a valid framework/slave off the authenticated list, forcing the valid framework/slave to re-authenticate and re-register. This could be used in a DoS attack. But how should we handle the scenario when the actual authenticated framework/slave sends an AuthenticateMessage that fails authentication? | 1 |
| 21 | MESOS-1114 | Authorize task/executor launches | NULL | 8 |
| 22 | MESOS-1119 | Allocator should make an allocation decision per slave instead of per framework/role. | Currently the Allocator::allocate() code loops through roles and frameworks (based on DRF sort) and allocates *all* slaves resources to the first framework. This logic should be a bit inversed. Instead, the slave should go through each slave, allocate it a role/framework and update the DRF shares. | 2 |
| 23 | MESOS-1120 | HTTP auth for CLI | Integrate HTTP auth into the CLI programs | 3 |
| 24 | MESOS-1127 | Implement the protobufs for the scheduler API | The default scheduler/executor interface and implementation in Mesos have a few drawbacks: (1) The interface is fairly high-level which makes it hard to do certain things, for example, handle events (callbacks) in batch. This can have a big impact on the performance of schedulers (for example, writing task updates that need to be persisted). (2) The implementation requires writing a lot of boilerplate JNI and native Python wrappers when adding additional API components. The plan is to provide a lower-level API that can easily be used to implement the higher-level API that is currently provided. This will also open the door to more easily building native-language Mesos libraries (i.e., not needing the C++ shim layer) and building new higher-level abstractions on top of the lower-level API. | 8 |
| 25 | MESOS-1143 | Add a TASK_ERROR task status. | During task validation we drop tasks that have errors and send TASK_LOST status updates. In most circumstances a framework will want to relaunch a task that has gone lost, and in the event the task is actually malformed (thus invalid) this will result in an infinite loop of sending a task and having it go lost. | 2 |
| 26 | MESOS-1148 | Add support for rate limiting slave removal | To safeguard against unforeseen bugs leading to widespread slave removal, it would be nice to allow for rate limiting of the decision to remove slaves and/or send TASK_LOST messages for tasks on those slaves. Ideally this would allow an operator to be notified soon enough to intervene before causing cluster impact. | 3 |
| 27 | MESOS-1195 | systemd.slice + cgroup enablement fails in multiple ways. | When attempting to configure mesos to use systemd slices on a 'rawhide/f21' machine, it fails creating the isolator: I0407 12:39:28.035354 14916 containerizer.cpp:180] Using isolation: cgroups/cpu,cgroups/mem Failed to create a containerizer: Could not create isolator cgroups/cpu: Failed to create isolator: The cpu subsystem is co-mounted at /sys/fs/cgroup/cpu with other subsytems ------ details ------ /sys/fs/cgroup total 0 drwxr-xr-x. 12 root root 280 Mar 18 08:47 . drwxr-xr-x. 6 root root 0 Mar 18 08:47 .. drwxr-xr-x. 2 root root 0 Mar 18 08:47 blkio lrwxrwxrwx. 1 root root 11 Mar 18 08:47 cpu -> cpu,cpuacct lrwxrwxrwx. 1 root root 11 Mar 18 08:47 cpuacct -> cpu,cpuacct drwxr-xr-x. 2 root root 0 Mar 18 08:47 cpu,cpuacct drwxr-xr-x. 2 root root 0 Mar 18 08:47 cpuset drwxr-xr-x. 2 root root 0 Mar 18 08:47 devices drwxr-xr-x. 2 root root 0 Mar 18 08:47 freezer drwxr-xr-x. 2 root root 0 Mar 18 08:47 hugetlb drwxr-xr-x. 3 root root 0 Apr 3 11:26 memory drwxr-xr-x. 2 root root 0 Mar 18 08:47 net_cls drwxr-xr-x. 2 root root 0 Mar 18 08:47 perf_event drwxr-xr-x. 4 root root 0 Mar 18 08:47 systemd | 3 |
| 28 | MESOS-1199 | Subprocess is "slow" -> gated by process::reap poll interval | Subprocess uses process::reap to wait on the subprocess pid and set the exit status. However, process::reap polls with a one second interval resulting in a delay up to the interval duration before the status future is set. This means if you need to wait for the subprocess to complete you get hit with E(delay) = 0.5 seconds, independent of the execution time. For example, the MesosContainerizer uses mesos-fetcher in a Subprocess to fetch the executor during launch. At Twitter we fetch a local file, i.e., a very fast operation, but the launch is blocked until the mesos-fetcher pid is reaped -> adding 0 to 1 seconds for every launch! The problem is even worse with a chain of short Subprocesses because after the first Subprocess completes you'll be synchronized with the reap interval and you'll see nearly the full interval before notification, i.e., 10 Subprocesses each of << 1 second duration with take ~10 seconds! This has become particularly apparent in some new tests I'm working on where test durations are now greatly extended with each taking several seconds. | 1 |
| 29 | MESOS-1219 | Master should disallow frameworks that reconnect after failover timeout. | When a scheduler reconnects after the failover timeout has exceeded, the framework id is usually reused because the scheduler doesn't know that the timeout exceeded and it is actually handled as a new framework. The /framework/:framework_id route of the Web UI doesn't handle those cases very well because its key is reused. It only shows the terminated one. Would it make sense to ignore the provided framework id when a scheduler reconnects to a terminated framework and generate a new id to make sure it's unique? | 2 |
| 30 | MESOS-1236 | stout's os module uses a mix of Try<Nothing> and bool returns | stout's os module should use Try<Nothing> for return values throughout. | 2 |
| 31 | MESOS-1237 | stout's os::ls should return a Try<> | stout's os::ls returns a list that can be empty - instead it should return a Try<list...> to be consistent. | 2 |
| 32 | MESOS-1303 | ExamplesTest.{TestFramework, NoExecutorFramework} flaky | I'm having trouble reproducing this but I did observe it once on my OSX system: {noformat} [==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from ExamplesTest [ RUN ] ExamplesTest.TestFramework ../../src/tests/script.cpp:81: Failure Failed test_framework_test.sh terminated with signal 'Abort trap: 6' [ FAILED ] ExamplesTest.TestFramework (953 ms) [ RUN ] ExamplesTest.NoExecutorFramework [ OK ] ExamplesTest.NoExecutorFramework (10162 ms) [----------] 2 tests from ExamplesTest (11115 ms total) [----------] Global test environment tear-down [==========] 2 tests from 1 test case ran. (11121 ms total) [ PASSED ] 1 test. [ FAILED ] 1 test, listed below: [ FAILED ] ExamplesTest.TestFramework {noformat} when investigating a failed make check for https://reviews.apache.org/r/20971/ {noformat} [----------] 6 tests from ExamplesTest [ RUN ] ExamplesTest.TestFramework [ OK ] ExamplesTest.TestFramework (8643 ms) [ RUN ] ExamplesTest.NoExecutorFramework tests/script.cpp:81: Failure Failed no_executor_framework_test.sh terminated with signal 'Aborted' [ FAILED ] ExamplesTest.NoExecutorFramework (7220 ms) [ RUN ] ExamplesTest.JavaFramework [ OK ] ExamplesTest.JavaFramework (11181 ms) [ RUN ] ExamplesTest.JavaException [ OK ] ExamplesTest.JavaException (5624 ms) [ RUN ] ExamplesTest.JavaLog [ OK ] ExamplesTest.JavaLog (6472 ms) [ RUN ] ExamplesTest.PythonFramework [ OK ] ExamplesTest.PythonFramework (14467 ms) [----------] 6 tests from ExamplesTest (53607 ms total) {noformat} | 1 |
| 33 | MESOS-1307 | Authorize offer allocations | When frameworks register or reregister they should authorize their roles. Split register framework / reregister framework. | 8 |
| 34 | MESOS-1316 | Implement decent unit test coverage for the mesos-fetcher tool | There are current no tests that cover the {{mesos-fetcher}} tool itself, and hence bugs like MESOS-1313 have accidentally slipped though. | 2 |
| 35 | MESOS-1332 | Improve Master and Slave metric names | As we move the metrics to a new endpoint, we should consider revisiting the names of some of the current metrics to make them clearer. It may also be worth considering changing some existing counter-style metrics to gauges. | 3 |
| 36 | MESOS-1339 | Add "per-framework-principal" counters for all messages from a scheduler on Master | Framework::principal is used identify one or more frameworks. If multiple frameworks use the same principal they'll have one counter showing their combined message count. | 3 |
| 37 | MESOS-1344 | Add flags support for JSON | NULL | 2 |
| 38 | MESOS-1347 | GarbageCollectorIntegrationTest.DiskUsage is flaky. | From Jenkins: https://builds.apache.org/job/Mesos-Ubuntu-distcheck/79/consoleFull {noformat} [ RUN ] GarbageCollectorIntegrationTest.DiskUsage Using temporary directory '/tmp/GarbageCollectorIntegrationTest_DiskUsage_pU3Ym7' I0507 03:27:38.775058 5758 leveldb.cpp:174] Opened db in 44.343989ms I0507 03:27:38.787498 5758 leveldb.cpp:181] Compacted db in 12.411065ms I0507 03:27:38.787533 5758 leveldb.cpp:196] Created db iterator in 4008ns I0507 03:27:38.787545 5758 leveldb.cpp:202] Seeked to beginning of db in 598ns I0507 03:27:38.787552 5758 leveldb.cpp:271] Iterated through 0 keys in the db in 173ns I0507 03:27:38.787564 5758 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0507 03:27:38.787858 5777 recover.cpp:425] Starting replica recovery I0507 03:27:38.788352 5793 master.cpp:267] Master 20140507-032738-453759884-58462-5758 (hemera.apache.org) started on 140.211.11.27:58462 I0507 03:27:38.788377 5793 master.cpp:304] Master only allowing authenticated frameworks to register I0507 03:27:38.788383 5793 master.cpp:309] Master only allowing authenticated slaves to register I0507 03:27:38.788389 5793 credentials.hpp:35] Loading credentials for authentication I0507 03:27:38.789064 5779 recover.cpp:451] Replica is in EMPTY status W0507 03:27:38.789115 5793 credentials.hpp:48] Failed to stat credentials file 'file:///tmp/GarbageCollectorIntegrationTest_DiskUsage_pU3Ym7/credentials': No such file or directory I0507 03:27:38.789489 5779 master.cpp:104] No whitelist given. Advertising offers for all slaves I0507 03:27:38.789531 5778 hierarchical_allocator_process.hpp:301] Initializing hierarchical allocator process with master : master@140.211.11.27:58462 I0507 03:27:38.791007 5788 replica.cpp:638] Replica in EMPTY status received a broadcasted recover request I0507 03:27:38.791177 5780 master.cpp:921] The newly elected leader is master@140.211.11.27:58462 with id 20140507-032738-453759884-58462-5758 I0507 03:27:38.791198 5780 master.cpp:931] Elected as the leading master! I0507 03:27:38.791205 5780 master.cpp:752] Recovering from registrar I0507 03:27:38.791251 5796 recover.cpp:188] Received a recover response from a replica in EMPTY status I0507 03:27:38.791323 5797 registrar.cpp:313] Recovering registrar I0507 03:27:38.792137 5795 recover.cpp:542] Updating replica status to STARTING I0507 03:27:38.807531 5781 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 15.124092ms I0507 03:27:38.807559 5781 replica.cpp:320] Persisted replica status to STARTING I0507 03:27:38.807621 5781 recover.cpp:451] Replica is in STARTING status I0507 03:27:38.809319 5799 replica.cpp:638] Replica in STARTING status received a broadcasted recover request I0507 03:27:38.809983 5795 recover.cpp:188] Received a recover response from a replica in STARTING status I0507 03:27:38.811204 5778 recover.cpp:542] Updating replica status to VOTING I0507 03:27:38.827595 5795 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 16.011355ms I0507 03:27:38.827627 5795 replica.cpp:320] Persisted replica status to VOTING I0507 03:27:38.827683 5795 recover.cpp:556] Successfully joined the Paxos group I0507 03:27:38.827775 5795 recover.cpp:440] Recover process terminated I0507 03:27:38.828966 5780 log.cpp:656] Attempting to start the writer I0507 03:27:38.831114 5782 replica.cpp:474] Replica received implicit promise request with proposal 1 I0507 03:27:38.847708 5782 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 16.573137ms I0507 03:27:38.847739 5782 replica.cpp:342] Persisted promised to 1 I0507 03:27:38.848141 5797 coordinator.cpp:230] Coordinator attemping to fill missing position I0507 03:27:38.849684 5790 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I0507 03:27:38.863777 5790 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 14.076775ms I0507 03:27:38.863801 5790 replica.cpp:676] Persisted action at 0 I0507 03:27:38.864915 5798 replica.cpp:508] Replica received write request for position 0 I0507 03:27:38.864949 5798 leveldb.cpp:436] Reading position from leveldb took 11807ns I0507 03:27:38.879945 5798 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 14.978446ms I0507 03:27:38.879976 5798 replica.cpp:676] Persisted action at 0 I0507 03:27:38.880491 5797 replica.cpp:655] Replica received learned notice for position 0 I0507 03:27:38.895969 5797 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 15.459949ms I0507 03:27:38.895992 5797 replica.cpp:676] Persisted action at 0 I0507 03:27:38.896003 5797 replica.cpp:661] Replica learned NOP action at position 0 I0507 03:27:38.896411 5783 log.cpp:672] Writer started with ending position 0 I0507 03:27:38.898058 5798 leveldb.cpp:436] Reading position from leveldb took 11910ns I0507 03:27:38.899749 5777 registrar.cpp:346] Successfully fetched the registry (0B) I0507 03:27:38.899766 5777 registrar.cpp:422] Attempting to update the 'registry' I0507 03:27:38.901458 5791 log.cpp:680] Attempting to append 137 bytes to the log I0507 03:27:38.901666 5780 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0507 03:27:38.902773 5783 replica.cpp:508] Replica received write request for position 1 I0507 03:27:38.916127 5783 leveldb.cpp:341] Persisting action (156 bytes) to leveldb took 13.225715ms I0507 03:27:38.916152 5783 replica.cpp:676] Persisted action at 1 I0507 03:27:38.916534 5790 replica.cpp:655] Replica received learned notice for position 1 I0507 03:27:38.928203 5790 leveldb.cpp:341] Persisting action (158 bytes) to leveldb took 11.652434ms I0507 03:27:38.928225 5790 replica.cpp:676] Persisted action at 1 I0507 03:27:38.928236 5790 replica.cpp:661] Replica learned APPEND action at position 1 I0507 03:27:38.928546 5790 registrar.cpp:479] Successfully updated 'registry' I0507 03:27:38.928642 5790 registrar.cpp:372] Successfully recovered registrar I0507 03:27:38.929044 5783 master.cpp:779] Recovered 0 slaves from the Registry (99B) ; allowing 10mins for slaves to re-register I0507 03:27:38.929502 5799 log.cpp:699] Attempting to truncate the log to 1 I0507 03:27:38.929888 5797 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0507 03:27:38.930161 5781 replica.cpp:508] Replica received write request for position 2 I0507 03:27:38.932977 5789 slave.cpp:140] Slave started on 56)@140.211.11.27:58462 I0507 03:27:38.932991 5789 credentials.hpp:35] Loading credentials for authentication W0507 03:27:38.933567 5789 credentials.hpp:48] Failed to stat credentials file 'file:///tmp/GarbageCollectorIntegrationTest_DiskUsage_A9Pxks/credential': No such file or directory I0507 03:27:38.933585 5789 slave.cpp:230] Slave using credential for: test-principal I0507 03:27:38.933765 5789 slave.cpp:243] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0507 03:27:38.933854 5789 slave.cpp:271] Slave hostname: hemera.apache.org I0507 03:27:38.933863 5789 slave.cpp:272] Slave checkpoint: false I0507 03:27:38.934239 5778 state.cpp:33] Recovering state from '/tmp/GarbageCollectorIntegrationTest_DiskUsage_A9Pxks/meta' I0507 03:27:38.934960 5792 status_update_manager.cpp:193] Recovering status update manager I0507 03:27:38.935123 5779 slave.cpp:2945] Finished recovery I0507 03:27:38.936998 5779 slave.cpp:526] New master detected at master@140.211.11.27:58462 I0507 03:27:38.937021 5779 slave.cpp:586] Authenticating with master master@140.211.11.27:58462 I0507 03:27:38.937077 5798 status_update_manager.cpp:167] New master detected at master@140.211.11.27:58462 I0507 03:27:38.937306 5779 slave.cpp:559] Detecting new master I0507 03:27:38.937335 5800 authenticatee.hpp:128] Creating new client SASL connection I0507 03:27:38.938030 5778 master.cpp:2798] Authenticating slave(56)@140.211.11.27:58462 I0507 03:27:38.938742 5783 authenticator.hpp:148] Creating new server SASL connection I0507 03:27:38.939312 5786 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0507 03:27:38.939340 5786 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0507 03:27:38.939390 5786 authenticator.hpp:254] Received SASL authentication start I0507 03:27:38.939553 5786 authenticator.hpp:342] Authentication requires more steps I0507 03:27:38.939592 5786 authenticatee.hpp:265] Received SASL authentication step I0507 03:27:38.939715 5786 authenticator.hpp:282] Received SASL authentication step I0507 03:27:38.939803 5786 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'hemera.apache.org' server FQDN: 'hemera.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0507 03:27:38.939821 5786 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0507 03:27:38.939831 5786 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0507 03:27:38.939841 5786 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'hemera.apache.org' server FQDN: 'hemera.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0507 03:27:38.939851 5786 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0507 03:27:38.939857 5786 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0507 03:27:38.939870 5786 authenticator.hpp:334] Authentication success I0507 03:27:38.939937 5786 authenticatee.hpp:305] Authentication success I0507 03:27:38.940016 5778 master.cpp:2838] Successfully authenticated slave(56)@140.211.11.27:58462 I0507 03:27:38.940449 5799 slave.cpp:643] Successfully authenticated with master master@140.211.11.27:58462 I0507 03:27:38.940513 5799 slave.cpp:872] Will retry registration in 5.176207635secs if necessary I0507 03:27:38.940625 5794 master.cpp:2134] Registering slave at slave(56)@140.211.11.27:58462 (hemera.apache.org) with id 20140507-032738-453759884-58462-5758-0 I0507 03:27:38.940800 5796 registrar.cpp:422] Attempting to update the 'registry' I0507 03:27:38.940850 5781 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 10.659152ms I0507 03:27:38.940871 5781 replica.cpp:676] Persisted action at 2 I0507 03:27:38.941843 5788 replica.cpp:655] Replica received learned notice for position 2 I0507 03:27:38.953193 5788 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 11.291343ms I0507 03:27:38.953258 5788 leveldb.cpp:399] Deleting ~1 keys from leveldb took 33725ns I0507 03:27:38.953274 5788 replica.cpp:676] Persisted action at 2 I0507 03:27:38.953282 5788 replica.cpp:661] Replica learned TRUNCATE action at position 2 I0507 03:27:38.953541 5797 log.cpp:680] Attempting to append 330 bytes to the log I0507 03:27:38.953614 5797 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I0507 03:27:38.954731 5789 replica.cpp:508] Replica received write request for position 3 I0507 03:27:38.965240 5789 leveldb.cpp:341] Persisting action (349 bytes) to leveldb took 10.489719ms I0507 03:27:38.965261 5789 replica.cpp:676] Persisted action at 3 I0507 03:27:38.966253 5780 replica.cpp:655] Replica received learned notice for position 3 I0507 03:27:38.977375 5780 leveldb.cpp:341] Persisting action (351 bytes) to leveldb took 11.098798ms I0507 03:27:38.977408 5780 replica.cpp:676] Persisted action at 3 I0507 03:27:38.977421 5780 replica.cpp:661] Replica learned APPEND action at position 3 I0507 03:27:38.977859 5792 registrar.cpp:479] Successfully updated 'registry' I0507 03:27:38.977926 5780 log.cpp:699] Attempting to truncate the log to 3 I0507 03:27:38.978060 5792 master.cpp:2174] Registered slave 20140507-032738-453759884-58462-5758-0 at slave(56)@140.211.11.27:58462 (hemera.apache.org) I0507 03:27:38.978112 5792 master.cpp:3283] Adding slave 20140507-032738-453759884-58462-5758-0 at slave(56)@140.211.11.27:58462 (hemera.apache.org) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0507 03:27:38.978134 5784 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 I0507 03:27:38.978508 5785 slave.cpp:676] Registered with master master@140.211.11.27:58462; given slave ID 20140507-032738-453759884-58462-5758-0 I0507 03:27:38.978631 5786 hierarchical_allocator_process.hpp:444] Added slave 20140507-032738-453759884-58462-5758-0 (hemera.apache.org) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] available) I0507 03:27:38.978677 5786 hierarchical_allocator_process.hpp:707] Performed allocation for slave 20140507-032738-453759884-58462-5758-0 in 5421ns I0507 03:27:38.979872 5796 replica.cpp:508] Replica received write request for position 4 I0507 03:27:38.982084 5758 sched.cpp:121] Version: 0.19.0 I0507 03:27:38.982213 5789 sched.cpp:217] New master detected at master@140.211.11.27:58462 I0507 03:27:38.982228 5789 sched.cpp:268] Authenticating with master master@140.211.11.27:58462 I0507 03:27:38.982347 5788 authenticatee.hpp:128] Creating new client SASL connection I0507 03:27:38.982676 5788 master.cpp:2798] Authenticating scheduler(59)@140.211.11.27:58462 I0507 03:27:38.983100 5788 authenticator.hpp:148] Creating new server SASL connection I0507 03:27:38.983294 5788 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0507 03:27:38.983312 5788 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0507 03:27:38.983360 5788 authenticator.hpp:254] Received SASL authentication start I0507 03:27:38.983505 5788 authenticator.hpp:342] Authentication requires more steps I0507 03:27:38.984220 5782 authenticatee.hpp:265] Received SASL authentication step I0507 03:27:38.984275 5782 authenticator.hpp:282] Received SASL authentication step I0507 03:27:38.984315 5782 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'hemera.apache.org' server FQDN: 'hemera.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0507 03:27:38.984347 5782 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0507 03:27:38.984359 5782 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0507 03:27:38.984370 5782 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'hemera.apache.org' server FQDN: 'hemera.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0507 03:27:38.984377 5782 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0507 03:27:38.984383 5782 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0507 03:27:38.984397 5782 authenticator.hpp:334] Authentication success I0507 03:27:38.984429 5782 authenticatee.hpp:305] Authentication success I0507 03:27:38.984469 5795 master.cpp:2838] Successfully authenticated scheduler(59)@140.211.11.27:58462 I0507 03:27:38.985110 5782 sched.cpp:342] Successfully authenticated with master master@140.211.11.27:58462 I0507 03:27:38.985133 5782 sched.cpp:461] Sending registration request to master@140.211.11.27:58462 I0507 03:27:38.985326 5795 master.cpp:980] Received registration request from scheduler(59)@140.211.11.27:58462 I0507 03:27:38.985357 5795 master.cpp:998] Registering framework 20140507-032738-453759884-58462-5758-0000 at scheduler(59)@140.211.11.27:58462 I0507 03:27:38.985424 5795 sched.cpp:392] Framework registered with 20140507-032738-453759884-58462-5758-0000 I0507 03:27:38.985471 5792 hierarchical_allocator_process.hpp:331] Added framework 20140507-032738-453759884-58462-5758-0000 I0507 03:27:38.985610 5795 sched.cpp:406] Scheduler::registered took 36702ns I0507 03:27:38.985646 5792 hierarchical_allocator_process.hpp:751] Offering cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140507-032738-453759884-58462-5758-0 to framework 20140507-032738-453759884-58462-5758-0000 I0507 03:27:38.985954 5792 hierarchical_allocator_process.hpp:687] Performed allocation for 1 slaves in 330895ns I0507 03:27:38.986001 5789 master.hpp:612] Adding offer 20140507-032738-453759884-58462-5758-0 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140507-032738-453759884-58462-5758-0 (hemera.apache.org) I0507 03:27:38.986090 5789 master.cpp:2747] Sending 1 offers to framework 20140507-032738-453759884-58462-5758-0000 I0507 03:27:38.986548 5792 sched.cpp:529] Scheduler::resourceOffers took 162873ns I0507 03:27:38.986721 5792 master.hpp:622] Removing offer 20140507-032738-453759884-58462-5758-0 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140507-032738-453759884-58462-5758-0 (hemera.apache.org) I0507 03:27:38.986781 5792 master.cpp:1812] Processing reply for offers: [ 20140507-032738-453759884-58462-5758-0 ] on slave 20140507-032738-453759884-58462-5758-0 at slave(56)@140.211.11.27:58462 (hemera.apache.org) for framework 20140507-032738-453759884-58462-5758-0000 I0507 03:27:38.986843 5792 master.hpp:584] Adding task 0 with resources cpus(*):2; mem(*):1024 on slave 20140507-032738-453759884-58462-5758-0 (hemera.apache.org) I0507 03:27:38.986876 5792 master.cpp:2922] Launching task 0 of framework 20140507-032738-453759884-58462-5758-0000 with resources cpus(*):2; mem(*):1024 on slave 20140507-032738-453759884-58462-5758-0 at slave(56)@140.211.11.27:58462 (hemera.apache.org) I0507 03:27:38.986981 5795 slave.cpp:906] Got assigned task 0 for framework 20140507-032738-453759884-58462-5758-0000 I0507 03:27:38.987180 5795 slave.cpp:1016] Launching task 0 for framework 20140507-032738-453759884-58462-5758-0000 I0507 03:27:38.987203 5787 hierarchical_allocator_process.hpp:546] Framework 20140507-032738-453759884-58462-5758-0000 left disk(*):1024; ports(*):[31000-32000] unused on slave 20140507-032738-453759884-58462-5758-0 I0507 03:27:38.987287 5787 hierarchical_allocator_process.hpp:589] Framework 20140507-032738-453759884-58462-5758-0000 filtered slave 20140507-032738-453759884-58462-5758-0 for 5secs I0507 03:27:38.991395 5795 exec.cpp:131] Version: 0.19.0 I0507 03:27:38.991497 5779 exec.cpp:181] Executor started at: executor(27)@140.211.11.27:58462 with pid 5758 I0507 03:27:38.991510 5795 slave.cpp:1126] Queuing task '0' for executor default of framework '20140507-032738-453759884-58462-5758-0000 I0507 03:27:38.991566 5795 slave.cpp:487] Successfully attached file '/tmp/GarbageCollectorIntegrationTest_DiskUsage_A9Pxks/slaves/20140507-032738-453759884-58462-5758-0/frameworks/20140507-032738-453759884-58462-5758-0000/executors/default/runs/de776bec-2822-4bbc-befc-eec40eb5f674' I0507 03:27:38.991595 5795 slave.cpp:2283] Monitoring executor 'default' of framework '20140507-032738-453759884-58462-5758-0000' in container 'de776bec-2822-4bbc-befc-eec40eb5f674' I0507 03:27:38.991778 5795 slave.cpp:1599] Got registration for executor 'default' of framework 20140507-032738-453759884-58462-5758-0000 I0507 03:27:38.991874 5795 slave.cpp:1718] Flushing queued task 0 for executor 'default' of framework 20140507-032738-453759884-58462-5758-0000 I0507 03:27:38.991935 5780 exec.cpp:205] Executor registered on slave 20140507-032738-453759884-58462-5758-0 I0507 03:27:38.993419 5796 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 13.489998ms I0507 03:27:38.993449 5796 replica.cpp:676] Persisted action at 4 I0507 03:27:38.994510 5777 replica.cpp:655] Replica received learned notice for position 4 I0507 03:27:... | 2 |
| 39 | MESOS-1358 | Show when the leading master was elected in the webui | This would be nice to have during debugging. | 1 |
| 40 | MESOS-1365 | SlaveRecoveryTest/0.MultipleFrameworks is flaky | --gtest_repeat=-1 --gtest_shuffle --gtest_break_on_failure {noformat} [ RUN ] SlaveRecoveryTest/0.MultipleFrameworks WARNING: Logging before InitGoogleLogging() is written to STDERR I0513 15:42:05.931761 4320 exec.cpp:131] Version: 0.19.0 I0513 15:42:05.936698 4340 exec.cpp:205] Executor registered on slave 20140513-154204-16842879-51872-13062-0 Registered executor on artoo Starting task 51991f97-f5fd-4905-ad0f-02668083af7c Forked command at 4367 sh -c 'sleep 1000' WARNING: Logging before InitGoogleLogging() is written to STDERR I0513 15:42:06.915061 4408 exec.cpp:131] Version: 0.19.0 I0513 15:42:06.931149 4435 exec.cpp:205] Executor registered on slave 20140513-154204-16842879-51872-13062-0 Registered executor on artoo Starting task eaf5d8d6-3a6c-4ee1-84c1-fae20fb1df83 sh -c 'sleep 1000' Forked command at 4439 I0513 15:42:06.998332 4340 exec.cpp:251] Received reconnect request from slave 20140513-154204-16842879-51872-13062-0 I0513 15:42:06.998414 4436 exec.cpp:251] Received reconnect request from slave 20140513-154204-16842879-51872-13062-0 I0513 15:42:07.006350 4437 exec.cpp:228] Executor re-registered on slave 20140513-154204-16842879-51872-13062-0 Re-registered executor on artoo I0513 15:42:07.027039 4337 exec.cpp:378] Executor asked to shutdown Shutting down Sending SIGTERM to process tree at pid 4367 Killing the following process trees: [ -+- 4367 sh -c sleep 1000 \--- 4368 sleep 1000 ] ../../src/tests/slave_recovery_tests.cpp:2807: Failure Value of: status1.get().state() Actual: TASK_FAILED Expected: TASK_KILLED Program received signal SIGSEGV, Segmentation fault. testing::UnitTest::AddTestPartResult (this=0x154dac0 <testing::UnitTest::GetInstance()::instance>, result_type=testing::TestPartResult::kFatalFailure, file_name=0xeb6b6c "../../src/tests/slave_recovery_tests.cpp", line_number=2807, message=..., os_stack_trace=...) at gmock-1.6.0/gtest/src/gtest.cc:3795 3795 *static_cast<volatile int*>(NULL) = 1; (gdb) bt #0 testing::UnitTest::AddTestPartResult (this=0x154dac0 <testing::UnitTest::GetInstance()::instance>, result_type=testing::TestPartResult::kFatalFailure, file_name=0xeb6b6c "../../src/tests/slave_recovery_tests.cpp", line_number=2807, message=..., os_stack_trace=...) at gmock-1.6.0/gtest/src/gtest.cc:3795 #1 0x0000000000df98b9 in testing::internal::AssertHelper::operator= (this=0x7fffffffb860, message=...) at gmock-1.6.0/gtest/src/gtest.cc:356 #2 0x0000000000cdfa57 in SlaveRecoveryTest_MultipleFrameworks_Test<mesos::internal::slave::MesosContainerizer>::TestBody (this=0x1954db0) at ../../src/tests/slave_recovery_tests.cpp:2807 #3 0x0000000000e22583 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x1954db0, method=&virtual testing::Test::TestBody(), location=0xed0af0 "the test body") at gmock-1.6.0/gtest/src/gtest.cc:2090 #4 0x0000000000e12467 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x1954db0, method=&virtual testing::Test::TestBody(), location=0xed0af0 "the test body") at gmock-1.6.0/gtest/src/gtest.cc:2126 #5 0x0000000000e010d5 in testing::Test::Run (this=0x1954db0) at gmock-1.6.0/gtest/src/gtest.cc:2161 #6 0x0000000000e01ceb in testing::TestInfo::Run (this=0x158cf80) at gmock-1.6.0/gtest/src/gtest.cc:2338 #7 0x0000000000e02387 in testing::TestCase::Run (this=0x158a880) at gmock-1.6.0/gtest/src/gtest.cc:2445 #8 0x0000000000e079ed in testing::internal::UnitTestImpl::RunAllTests (this=0x1558b40) at gmock-1.6.0/gtest/src/gtest.cc:4237 #9 0x0000000000e1ec83 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x1558b40, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0xe07700 <testing::internal::UnitTestImpl::RunAllTests()>, location=0xed1219 "auxiliary test code (environments or event listeners)") at gmock-1.6.0/gtest/src/gtest.cc:2090 #10 0x0000000000e14217 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x1558b40, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0xe07700 <testing::internal::UnitTestImpl::RunAllTests()>, location=0xed1219 "auxiliary test code (environments or event listeners)") at gmock-1.6.0/gtest/src/gtest.cc:2126 #11 0x0000000000e076d7 in testing::UnitTest::Run (this=0x154dac0 <testing::UnitTest::GetInstance()::instance>) at gmock-1.6.0/gtest/src/gtest.cc:3872 #12 0x0000000000b99887 in main (argc=1, argv=0x7fffffffd9f8) at ../../src/tests/main.cpp:107 (gdb) frame 2 #2 0x0000000000cdfa57 in SlaveRecoveryTest_MultipleFrameworks_Test<mesos::internal::slave::MesosContainerizer>::TestBody (this=0x1954db0) at ../../src/tests/slave_recovery_tests.cpp:2807 2807 ASSERT_EQ(TASK_KILLED, status1.get().state()); (gdb) p status1 $1 = {data = {<std::__shared_ptr<process::Future<mesos::TaskStatus>::Data, 2>> = {_M_ptr = 0x1963140, _M_refcount = {_M_pi = 0x198a620}}, <No data fields>}} (gdb) p status1.get() $2 = (const mesos::TaskStatus &) @0x7fffdc5bf5f0: {<google::protobuf::Message> = {<google::protobuf::MessageLite> = {_vptr$MessageLite = 0x7ffff74bc940 <vtable for mesos::TaskStatus+16>}, <No data fields>}, static kTaskIdFieldNumber = 1, static kStateFieldNumber = 2, static kMessageFieldNumber = 4, static kDataFieldNumber = 3, static kSlaveIdFieldNumber = 5, static kTimestampFieldNumber = 6, _unknown_fields_ = {fields_ = 0x0}, task_id_ = 0x7fffdc5ce9a0, message_ = 0x7fffdc5f5880, data_ = 0x154b4b0 <google::protobuf::internal::kEmptyString>, slave_id_ = 0x7fffdc59c4f0, timestamp_ = 1429688582.046252, state_ = 3, _cached_size_ = 0, _has_bits_ = {55}, static default_instance_ = 0x0} (gdb) p status1.get().state() $3 = mesos::TASK_FAILED (gdb) list 2802 // Kill task 1. 2803 driver1.killTask(task1.task_id()); 2804 2805 // Wait for TASK_KILLED update. 2806 AWAIT_READY(status1); 2807 ASSERT_EQ(TASK_KILLED, status1.get().state()); 2808 2809 // Kill task 2. 2810 driver2.killTask(task2.task_id()); 2811 {noformat} | 1 |
| 41 | MESOS-1371 | Expose libprocess queue length from scheduler driver to metrics endpoint | We expose the master's event queue length and we should do the same for the scheduler driver. | 1 |
| 42 | MESOS-1373 | Keep track of the principals for authenticated pids in Master. | Need to add a 'principal' field to FrameworkInfo and verify if the Framework has the claimed principal during registration. | 3 |
| 43 | MESOS-1374 | Verify static libprocess scheduler port works with Mesos Master | NULL | 5 |
| 44 | MESOS-1392 | Failure when znode is removed before we can read its contents. | Looks like the following can occur when a znode goes away right before we can read it's contents: {noformat: title=Slave exit} I0520 16:33:45.721727 29155 group.cpp:382] Trying to create path '/home/mesos/test/master' in ZooKeeper I0520 16:33:48.600837 29155 detector.cpp:134] Detected a new leader: (id='2617') I0520 16:33:48.601428 29147 group.cpp:655] Trying to get '/home/mesos/test/master/info_0000002617' in ZooKeeper Failed to detect a master: Failed to get data for ephemeral node '/home/mesos/test/master/info_0000002617' in ZooKeeper: no node Slave Exit Status: 1 {noformat} | 3 |
| 45 | MESOS-1393 | Write parser for perf output. | 1. Should support output from pid and cgroup targets. 2. Should support output for the same events from >= 1 cgroup 3. Should return as PerfStatistics protobuf. | 3 |
| 46 | MESOS-1394 | Test different versions of perf | Test across different kernel versions (at least 2.6.XX and 3.X) and across different distributions. Test input flags and parsing output. | 3 |
| 47 | MESOS-1395 | Test perf isolator for slave roll forward/roll back | Test that changes to add/remove perf isolator will be handled through slave recovery, e.g., containers started without the perf isolator continue to report resource statistics and containers started with the perf isolator will include perf statistics. | 2 |
| 48 | MESOS-1396 | Introduce a PerfStatistics protobuf | Field names from `perf list` normalized to convert hyphens to underscores and down-cased. Start with just the hardware and software events, not raw hardware, breakpoints or tracepoints, All fields should be optional. Include as an optional field to ResourceStatistics. | 2 |
| 49 | MESOS-1397 | Rename ResourceStatistics for containers | Rename ContainerStatistics which includes optional ResourceStatistics and optional PerfStatistics. | 8 |
| 50 | MESOS-1398 | Document perf isolator flags | Document interval, duration and the event flags. Document event name normalization for the protobuf. | 1 |
| 51 | MESOS-1410 | Keep terminal unacknowledged tasks in the master's state. | Once we are sending acknowledgments through the master as per MESOS-1409, we need to keep terminal tasks that are *unacknowledged* in the Master's memory. This will allow us to identify these tasks to frameworks when we haven't yet forwarded them an update. Without this, we're susceptible to MESOS-1389. | 5 |
| 52 | MESOS-1424 | Mesos tests should not rely on echo | Triggered by MESOS-1413 I would like to propose changing our tests to not rely on {{echo}} but to use {{printf}} instead. This seems to be useful as {{echo}} is introducing an extra linefeed after the supplied string whereas {{printf}} does not. The {{-n}} switch preventing that extra linefeed is unfortunately not portable - it is not supported by the builtin {{echo}} of the BSD / OSX {{/bin/sh}}. | 1 |
| 53 | MESOS-1425 | LogZooKeeperTest.WriteRead test is flaky | {code} [ RUN ] LogZooKeeperTest.WriteRead I0527 23:23:48.286031 1352 zookeeper_test_server.cpp:158] Started ZooKeeperTestServer on port 39446 I0527 23:23:48.293916 1352 log_tests.cpp:1945] Using temporary directory '/tmp/LogZooKeeperTest_WriteRead_Vyty8g' I0527 23:23:48.296430 1352 leveldb.cpp:176] Opened db in 2.459713ms I0527 23:23:48.296740 1352 leveldb.cpp:183] Compacted db in 286843ns I0527 23:23:48.296761 1352 leveldb.cpp:198] Created db iterator in 3083ns I0527 23:23:48.296772 1352 leveldb.cpp:204] Seeked to beginning of db in 4541ns I0527 23:23:48.296777 1352 leveldb.cpp:273] Iterated through 0 keys in the db in 87ns I0527 23:23:48.296788 1352 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0527 23:23:48.297499 1383 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 505340ns I0527 23:23:48.297513 1383 replica.cpp:320] Persisted replica status to VOTING I0527 23:23:48.299492 1352 leveldb.cpp:176] Opened db in 1.73582ms I0527 23:23:48.299773 1352 leveldb.cpp:183] Compacted db in 263937ns I0527 23:23:48.299793 1352 leveldb.cpp:198] Created db iterator in 7494ns I0527 23:23:48.299806 1352 leveldb.cpp:204] Seeked to beginning of db in 235ns I0527 23:23:48.299813 1352 leveldb.cpp:273] Iterated through 0 keys in the db in 93ns I0527 23:23:48.299821 1352 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0527 23:23:48.300503 1380 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 492309ns I0527 23:23:48.300516 1380 replica.cpp:320] Persisted replica status to VOTING I0527 23:23:48.302500 1352 leveldb.cpp:176] Opened db in 1.793829ms I0527 23:23:48.303642 1352 leveldb.cpp:183] Compacted db in 1.123929ms I0527 23:23:48.303669 1352 leveldb.cpp:198] Created db iterator in 5865ns I0527 23:23:48.303689 1352 leveldb.cpp:204] Seeked to beginning of db in 8811ns I0527 23:23:48.303705 1352 leveldb.cpp:273] Iterated through 1 keys in the db in 9545ns I0527 23:23:48.303715 1352 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned 2014-05-27 23:23:48,303:1352(0x2b1173a29700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2014-05-27 23:23:48,303:1352(0x2b1173a29700):ZOO_INFO@log_env@716: Client environment:host.name=minerva 2014-05-27 23:23:48,303:1352(0x2b1173a29700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2014-05-27 23:23:48,303:1352(0x2b1173a29700):ZOO_INFO@log_env@724: Client environment:os.arch=3.2.0-57-generic 2014-05-27 23:23:48,303:1352(0x2b1173a29700):ZOO_INFO@log_env@725: Client environment:os.version=#87-Ubuntu SMP Tue Nov 12 21:35:10 UTC 2013 2014-05-27 23:23:48,303:1352(0x2b1173e2b700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@log_env@716: Client environment:host.name=minerva 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@log_env@724: Client environment:os.arch=3.2.0-57-generic 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@log_env@725: Client environment:os.version=#87-Ubuntu SMP Tue Nov 12 21:35:10 UTC 2013 2014-05-27 23:23:48,304:1352(0x2b1173a29700):ZOO_INFO@log_env@733: Client environment:user.name=(null) I0527 23:23:48.303988 1380 log.cpp:238] Attempting to join replica to ZooKeeper group 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@log_env@733: Client environment:user.name=(null) 2014-05-27 23:23:48,304:1352(0x2b1173a29700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins I0527 23:23:48.304198 1385 recover.cpp:425] Starting replica recovery 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins 2014-05-27 23:23:48,304:1352(0x2b1173a29700):ZOO_INFO@log_env@753: Client environment:user.dir=/tmp/LogZooKeeperTest_WriteRead_Vyty8g 2014-05-27 23:23:48,304:1352(0x2b1173a29700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:39446 sessionTimeout=5000 watcher=0x2b11708e98d0 sessionId=0 sessionPasswd=<null> context=0x2b118002f4e0 flags=0 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@log_env@753: Client environment:user.dir=/tmp/LogZooKeeperTest_WriteRead_Vyty8g I0527 23:23:48.304352 1385 recover.cpp:451] Replica is in VOTING status 2014-05-27 23:23:48,304:1352(0x2b1173e2b700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:39446 sessionTimeout=5000 watcher=0x2b11708e98d0 sessionId=0 sessionPasswd=<null> context=0x2b1198015ca0 flags=0 I0527 23:23:48.304417 1385 recover.cpp:440] Recover process terminated 2014-05-27 23:23:48,304:1352(0x2b12897b8700):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:39446] 2014-05-27 23:23:48,304:1352(0x2b12891b5700):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:39446] I0527 23:23:48.311262 1352 leveldb.cpp:176] Opened db in 7.261703ms 2014-05-27 23:23:48,311:1352(0x2b12897b8700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:39446], sessionId=0x1463fff34bd0000, negotiated timeout=6000 I0527 23:23:48.312379 1381 group.cpp:310] Group process ((614)@67.195.138.8:35151) connected to ZooKeeper I0527 23:23:48.312407 1381 group.cpp:784] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I0527 23:23:48.312417 1381 group.cpp:382] Trying to create path '/log' in ZooKeeper I0527 23:23:48.312422 1352 leveldb.cpp:183] Compacted db in 1.119843ms I0527 23:23:48.312505 1352 leveldb.cpp:198] Created db iterator in 3901ns I0527 23:23:48.312526 1352 leveldb.cpp:204] Seeked to beginning of db in 7398ns I0527 23:23:48.312541 1352 leveldb.cpp:273] Iterated through 1 keys in the db in 6345ns I0527 23:23:48.312553 1352 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned 2014-05-27 23:23:48,312:1352(0x2b1173627700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2014-05-27 23:23:48,312:1352(0x2b1173627700):ZOO_INFO@log_env@716: Client environment:host.name=minerva 2014-05-27 23:23:48,312:1352(0x2b1173627700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2014-05-27 23:23:48,312:1352(0x2b1173627700):ZOO_INFO@log_env@724: Client environment:os.arch=3.2.0-57-generic 2014-05-27 23:23:48,312:1352(0x2b1173627700):ZOO_INFO@log_env@725: Client environment:os.version=#87-Ubuntu SMP Tue Nov 12 21:35:10 UTC 2013 2014-05-27 23:23:48,312:1352(0x2b1173627700):ZOO_INFO@log_env@733: Client environment:user.name=(null) 2014-05-27 23:23:48,312:1352(0x2b12891b5700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:39446], sessionId=0x1463fff34bd0001, negotiated timeout=6000 2014-05-27 23:23:48,313:1352(0x2b1173627700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins 2014-05-27 23:23:48,313:1352(0x2b1173627700):ZOO_INFO@log_env@753: Client environment:user.dir=/tmp/LogZooKeeperTest_WriteRead_Vyty8g 2014-05-27 23:23:48,313:1352(0x2b1173627700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:39446 sessionTimeout=5000 watcher=0x2b11708e98d0 sessionId=0 sessionPasswd=<null> context=0x2b119001fd20 flags=0 I0527 23:23:48.313247 1380 group.cpp:310] Group process ((616)@67.195.138.8:35151) connected to ZooKeeper I0527 23:23:48.313266 1380 group.cpp:784] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0) I0527 23:23:48.313273 1380 group.cpp:382] Trying to create path '/log' in ZooKeeper 2014-05-27 23:23:48,313:1352(0x2b12889b0700):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:39446] 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@716: Client environment:host.name=minerva 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@724: Client environment:os.arch=3.2.0-57-generic 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@725: Client environment:os.version=#87-Ubuntu SMP Tue Nov 12 21:35:10 UTC 2013 I0527 23:23:48.313436 1387 log.cpp:238] Attempting to join replica to ZooKeeper group 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@733: Client environment:user.name=(null) 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@log_env@753: Client environment:user.dir=/tmp/LogZooKeeperTest_WriteRead_Vyty8g 2014-05-27 23:23:48,313:1352(0x2b1173828700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:39446 sessionTimeout=5000 watcher=0x2b11708e98d0 sessionId=0 sessionPasswd=<null> context=0x2b1190011ea0 flags=0 I0527 23:23:48.313601 1387 recover.cpp:425] Starting replica recovery I0527 23:23:48.313721 1382 recover.cpp:451] Replica is in VOTING status I0527 23:23:48.313794 1382 recover.cpp:440] Recover process terminated 2014-05-27 23:23:48,313:1352(0x2b1288bb1700):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:39446] I0527 23:23:48.313973 1383 log.cpp:656] Attempting to start the writer 2014-05-27 23:23:48,315:1352(0x2b12889b0700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:39446], sessionId=0x1463fff34bd0002, negotiated timeout=6000 I0527 23:23:48.315682 1387 group.cpp:310] Group process ((619)@67.195.138.8:35151) connected to ZooKeeper 2014-05-27 23:23:48,315:1352(0x2b1288bb1700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:39446], sessionId=0x1463fff34bd0003, negotiated timeout=6000 I0527 23:23:48.315709 1387 group.cpp:784] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I0527 23:23:48.315738 1387 group.cpp:382] Trying to create path '/log' in ZooKeeper I0527 23:23:48.315964 1386 group.cpp:310] Group process ((621)@67.195.138.8:35151) connected to ZooKeeper I0527 23:23:48.315981 1386 group.cpp:784] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0) I0527 23:23:48.315989 1386 group.cpp:382] Trying to create path '/log' in ZooKeeper I0527 23:23:48.317881 1385 network.hpp:423] ZooKeeper group memberships changed I0527 23:23:48.317937 1381 group.cpp:655] Trying to get '/log/0000000000' in ZooKeeper I0527 23:23:48.318205 1382 network.hpp:423] ZooKeeper group memberships changed I0527 23:23:48.318317 1383 group.cpp:655] Trying to get '/log/0000000000' in ZooKeeper I0527 23:23:48.319154 1382 network.hpp:461] ZooKeeper group PIDs: { log-replica(22)@67.195.138.8:35151 } I0527 23:23:48.319541 1386 network.hpp:461] ZooKeeper group PIDs: { log-replica(22)@67.195.138.8:35151 } I0527 23:23:48.319851 1381 replica.cpp:474] Replica received implicit promise request with proposal 1 I0527 23:23:48.319905 1387 replica.cpp:474] Replica received implicit promise request with proposal 1 I0527 23:23:48.319907 1384 network.hpp:423] ZooKeeper group memberships changed I0527 23:23:48.320091 1385 group.cpp:655] Trying to get '/log/0000000000' in ZooKeeper I0527 23:23:48.320384 1383 network.hpp:423] ZooKeeper group memberships changed I0527 23:23:48.320441 1381 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 568396ns I0527 23:23:48.320456 1384 group.cpp:655] Trying to get '/log/0000000000' in ZooKeeper I0527 23:23:48.320461 1381 replica.cpp:342] Persisted promised to 1 I0527 23:23:48.320446 1387 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 516015ns I0527 23:23:48.320497 1387 replica.cpp:342] Persisted promised to 1 I0527 23:23:48.320814 1383 coordinator.cpp:230] Coordinator attemping to fill missing position I0527 23:23:48.321050 1384 group.cpp:655] Trying to get '/log/0000000001' in ZooKeeper I0527 23:23:48.321063 1385 group.cpp:655] Trying to get '/log/0000000001' in ZooKeeper I0527 23:23:48.321341 1387 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I0527 23:23:48.321375 1381 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I0527 23:23:48.321506 1387 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 89us I0527 23:23:48.321530 1387 replica.cpp:676] Persisted action at 0 I0527 23:23:48.321584 1381 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 122910ns I0527 23:23:48.321602 1381 replica.cpp:676] Persisted action at 0 I0527 23:23:48.321775 1383 network.hpp:461] ZooKeeper group PIDs: { log-replica(22)@67.195.138.8:35151, log-replica(23)@67.195.138.8:35151 } I0527 23:23:48.321961 1381 replica.cpp:508] Replica received write request for position 0 I0527 23:23:48.321984 1381 leveldb.cpp:438] Reading position from leveldb took 7813ns I0527 23:23:48.322064 1380 network.hpp:461] ZooKeeper group PIDs: { log-replica(22)@67.195.138.8:35151, log-replica(23)@67.195.138.8:35151 } I0527 23:23:48.322073 1381 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 78683ns I0527 23:23:48.322077 1383 replica.cpp:508] Replica received write request for position 0 I0527 23:23:48.322084 1381 replica.cpp:676] Persisted action at 0 I0527 23:23:48.322111 1383 leveldb.cpp:438] Reading position from leveldb took 17416ns I0527 23:23:48.322330 1383 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 157199ns I0527 23:23:48.322345 1383 replica.cpp:676] Persisted action at 0 I0527 23:23:48.322522 1386 replica.cpp:655] Replica received learned notice for position 0 I0527 23:23:48.322523 1382 replica.cpp:655] Replica received learned notice for position 0 I0527 23:23:48.322638 1386 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 86907ns I0527 23:23:48.322661 1386 replica.cpp:676] Persisted action at 0 I0527 23:23:48.322670 1386 replica.cpp:661] Replica learned NOP action at position 0 I0527 23:23:48.322682 1382 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 85031ns I0527 23:23:48.322693 1382 replica.cpp:676] Persisted action at 0 I0527 23:23:48.322700 1382 replica.cpp:661] Replica learned NOP action at position 0 I0527 23:23:48.322790 1380 log.cpp:672] Writer started with ending position 0 I0527 23:23:48.322898 1380 log.cpp:680] Attempting to append 11 bytes to the log I0527 23:23:48.322978 1383 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0527 23:23:48.323122 1380 replica.cpp:508] Replica received write request for position 1 I0527 23:23:48.323158 1381 replica.cpp:508] Replica received write request for position 1 I0527 23:23:48.323202 1380 leveldb.cpp:343] Persisting action (27 bytes) to leveldb took 66527ns I0527 23:23:48.323215 1380 replica.cpp:676] Persisted action at 1 I0527 23:23:48.323238 1381 leveldb.cpp:343] Persisting action (27 bytes) to leveldb took 67074ns I0527 23:23:48.323252 1381 replica.cpp:676] Persisted action at 1 I0527 23:23:48.323354 1380 replica.cpp:655] Replica received learned notice for position 1 I0527 23:23:48.323362 1382 replica.cpp:655] Replica received learned notice for position 1 I0527 23:23:48.323443 1380 leveldb.cpp:343] Persisting action (29 bytes) to leveldb took 77398ns I0527 23:23:48.323461 1380 replica.cpp:676] Persisted action at 1 I0527 23:23:48.323463 1382 leveldb.cpp:343] Persisting action (29 bytes) to leveldb took 90567ns I0527 23:23:48.323467 1380 replica.cpp:661] Replica learned APPEND action at position 1 I0527 23:23:48.323477 1382 replica.cpp:676] Persisted action at 1 I0527 23:23:48.323484 1382 replica.cpp:661] Replica learned APPEND action at position 1 I0527 23:23:48.323729 1380 leveldb.cpp:438] Reading position from leveldb took 7224ns 2014-05-27 23:23:48,324:1352(0x2b1173c2a700):ZOO_INFO@zookeeper_close@2505: Closing zookeeper sessionId=0x1463fff34bd0003 to [127.0.0.1:39446] 2014-05-27 23:23:48,324:1352(0x2b117301ff80):ZOO_INFO@zookeeper_close@2505: Closing zookeeper sessionId=0x1463fff34bd0002 to [127.0.0.1:39446] I0527 23:23:48.326591 1386 network.hpp:423] ZooKeeper group memberships changed I0527 23:23:48.326690 1382 group.cpp:655] Trying to get '/log/0000000000' in ZooKeeper I0527 23:23:48.327450 1384 network.hpp:461] ZooKeeper group PIDs: { log-replica(22)@67.195.138.8:35151 } 2014-05-27 23:23:48,446:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:23:51,782:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:23:55,118:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client I0527 23:23:57.002908 1381 network.hpp:423] ZooKeeper group memberships changed I0527 23:23:57.003042 1381 network.hpp:461] ZooKeeper group PIDs: { } 2014-05-27 23:23:58,455:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:01,791:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:05,127:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:08,464:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:11,800:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:15,136:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:18,473:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:21,809:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:25,146:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:28,482:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:31,818:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:35,155:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:38,491:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-05-27 23:24:41,827:1352(0x2b12bc401700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:51020] zk retc... | 1 |
| 54 | MESOS-1443 | Create a protobuf for framework rate limit configuration and load it as JSON through master flags | NULL | 2 |
| 55 | MESOS-1444 | Integrate rate limiter into the master | NULL | 5 |
| 56 | MESOS-1445 | Add new tests for framework rate limiting | NULL | 3 |
| 57 | MESOS-1459 | Build failure: Ubuntu 13.10/clang due to missing virtual destructor | In file included from launcher/main.cpp:19: In file included from ./launcher/launcher.hpp:24: In file included from ../3rdparty/libprocess/include/process/future.hpp:23: ../3rdparty/libprocess/include/process/owned.hpp:188:5: error: delete called on 'mesos::internal::launcher::Operation' that is abstract but has non-virtual destructor [-Werror,-Wdelete-non-virtual-dtor] delete t; ^ /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/shared_ptr_base.h:456:8: note: in instantiation of member function 'process::Owned<mesos::internal::launcher::Operation>::Data::~Data' requested here delete __p; ^ /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/shared_ptr_base.h:768:24: note: in instantiation of function template specialization 'std::__shared_count<2>::__shared_count<process::Owned<mesos::internal::launcher::Operation>::Data *>' requested here : _M_ptr(__p), _M_refcount(__p) ^ /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/shared_ptr_base.h:919:4: note: in instantiation of function template specialization 'std::__shared_ptr<process::Owned<mesos::internal::launcher::Operation>::Data, 2>::__shared_ptr<process::Owned<mesos::internal::launcher::Operation>::Data>' requested here __shared_ptr(__p).swap(*this); ^ ../3rdparty/libprocess/include/process/owned.hpp:68:10: note: in instantiation of function template specialization 'std::__shared_ptr<process::Owned<mesos::internal::launcher::Operation>::Data, 2>::reset<process::Owned<mesos::internal::launcher::Operation>::Data>' requested here data.reset(new Data(t)); ^ ./launcher/launcher.hpp:101:7: note: in instantiation of member function 'process::Owned<mesos::internal::launcher::Operation>::Owned' requested here add(process::Owned<Operation>(new T())); ^ launcher/main.cpp:26:3: note: in instantiation of function template specialization 'mesos::internal::launcher::add<mesos::internal::launcher::ShellOperation>' requested here launcher::add<launcher::ShellOperation>(); ^ 1 error generated. | 1 |
| 58 | MESOS-1466 | Race between executor exited event and launch task can cause overcommit of resources | The following sequence of events can cause an overcommit --> Launch task is called for a task whose executor is already running --> Executor's resources are not accounted for on the master --> Executor exits and the event is enqueued behind launch tasks on the master --> Master sends the task to the slave which needs to commit for resources for task and the (new) executor. --> Master processes the executor exited event and re-offers the executor's resources causing an overcommit of resources. | 8 |
| 59 | MESOS-1469 | No output from review bot on timeout | When the mesos review build times out, likely due to a long-running failing test, we have no output to debug. We should find a way to stream the output from the build instead of waiting for the build to finish. | 1 |
| 60 | MESOS-1471 | Document replicated log design/internals | The replicated log could benefit from some documentation. In particular, how does it work? What do operators need to know? Possibly there is some overlap with our future maintenance documentation in MESOS-1470. I believe [~jieyu] has some unpublished work that could be leveraged here! | 5 |
| 61 | MESOS-1472 | Improve child exit if slave dies during executor launch in MC | When restarting many slaves there's a reasonable chance that a slave will be restarted between the fork and exec stages of launching an executor in the MesosContainerizer. The forked child correctly detects this however rather than abort it should safely log and then exit non-zero cleanly. | 1 |
| 62 | MESOS-1518 | Update Rate Limiting Design doc to reflect the latest changes | - Usage - Design - Implementation Notes | 2 |
| 63 | MESOS-1527 | Choose containerizer at runtime | Currently you have to choose the containerizer at mesos-slave start time via the --isolation option. I'd like to be able to specify the containerizer in the request to launch the job. This could be specified by a new "Provider" field in the ContainerInfo proto buf. | 3 |
| 64 | MESOS-1529 | Handle a network partition between Master and Slave | If a network partition occurs between a Master and Slave, the Master will remove the Slave (as it fails health check) and mark the tasks being run there as LOST. However, the Slave is not aware that it has been removed so the tasks will continue to run. (To clarify a little bit: neither the master nor the slave receives 'exited' event, indicating that the connection between the master and slave is not closed). There are at least two possible approaches to solving this issue: 1. Introduce a health check from Slave to Master so they have a consistent view of a network partition. We may still see this issue should a one-way connection error occur. 2. Be less aggressive about marking tasks and Slaves as lost. Wait until the Slave reappears and reconcile then. We'd still need to mark Slaves and tasks as potentially lost (zombie state) but maybe the Scheduler can make a more intelligent decision. | 5 |
| 65 | MESOS-1545 | SlaveRecoveryTest/0.MultipleFrameworks is flaky | {code} [ RUN ] SlaveRecoveryTest/0.MultipleFrameworks Using temporary directory '/tmp/SlaveRecoveryTest_0_MultipleFrameworks_6dJqxr' I0626 00:04:39.557339 5450 leveldb.cpp:176] Opened db in 179.857593ms I0626 00:04:39.565433 5450 leveldb.cpp:183] Compacted db in 8.071041ms I0626 00:04:39.565457 5450 leveldb.cpp:198] Created db iterator in 4065ns I0626 00:04:39.565466 5450 leveldb.cpp:204] Seeked to beginning of db in 596ns I0626 00:04:39.565474 5450 leveldb.cpp:273] Iterated through 0 keys in the db in 396ns I0626 00:04:39.565490 5450 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0626 00:04:39.565827 5476 recover.cpp:425] Starting replica recovery I0626 00:04:39.566033 5474 recover.cpp:451] Replica is in EMPTY status I0626 00:04:39.566504 5474 replica.cpp:638] Replica in EMPTY status received a broadcasted recover request I0626 00:04:39.566686 5477 recover.cpp:188] Received a recover response from a replica in EMPTY status I0626 00:04:39.566905 5472 recover.cpp:542] Updating replica status to STARTING I0626 00:04:39.568307 5471 master.cpp:288] Master 20140626-000439-1032504131-55423-5450 (juno.apache.org) started on 67.195.138.61:55423 I0626 00:04:39.568332 5471 master.cpp:325] Master only allowing authenticated frameworks to register I0626 00:04:39.568339 5471 master.cpp:330] Master only allowing authenticated slaves to register I0626 00:04:39.568348 5471 credentials.hpp:35] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_MultipleFrameworks_6dJqxr/credentials' I0626 00:04:39.568461 5471 master.cpp:356] Authorization enabled I0626 00:04:39.568739 5478 master.cpp:122] No whitelist given. Advertising offers for all slaves I0626 00:04:39.568814 5475 hierarchical_allocator_process.hpp:301] Initializing hierarchical allocator process with master : master@67.195.138.61:55423 I0626 00:04:39.569206 5478 master.cpp:1122] The newly elected leader is master@67.195.138.61:55423 with id 20140626-000439-1032504131-55423-5450 I0626 00:04:39.569223 5478 master.cpp:1135] Elected as the leading master! I0626 00:04:39.569231 5478 master.cpp:953] Recovering from registrar I0626 00:04:39.569286 5475 registrar.cpp:313] Recovering registrar I0626 00:04:39.600639 5477 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 33.682136ms I0626 00:04:39.600661 5477 replica.cpp:320] Persisted replica status to STARTING I0626 00:04:39.600790 5476 recover.cpp:451] Replica is in STARTING status I0626 00:04:39.601184 5474 replica.cpp:638] Replica in STARTING status received a broadcasted recover request I0626 00:04:39.601274 5477 recover.cpp:188] Received a recover response from a replica in STARTING status I0626 00:04:39.601465 5471 recover.cpp:542] Updating replica status to VOTING I0626 00:04:39.610605 5471 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 9.076262ms I0626 00:04:39.610638 5471 replica.cpp:320] Persisted replica status to VOTING I0626 00:04:39.610683 5471 recover.cpp:556] Successfully joined the Paxos group I0626 00:04:39.610780 5471 recover.cpp:440] Recover process terminated I0626 00:04:39.610946 5474 log.cpp:656] Attempting to start the writer I0626 00:04:39.611486 5475 replica.cpp:474] Replica received implicit promise request with proposal 1 I0626 00:04:39.618924 5475 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 7.418789ms I0626 00:04:39.618942 5475 replica.cpp:342] Persisted promised to 1 I0626 00:04:39.619220 5476 coordinator.cpp:230] Coordinator attemping to fill missing position I0626 00:04:39.619763 5476 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I0626 00:04:39.627267 5476 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 7.485492ms I0626 00:04:39.627295 5476 replica.cpp:676] Persisted action at 0 I0626 00:04:39.627822 5473 replica.cpp:508] Replica received write request for position 0 I0626 00:04:39.627861 5473 leveldb.cpp:438] Reading position from leveldb took 17132ns I0626 00:04:39.635592 5473 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 7.714322ms I0626 00:04:39.635612 5473 replica.cpp:676] Persisted action at 0 I0626 00:04:39.635797 5473 replica.cpp:655] Replica received learned notice for position 0 I0626 00:04:39.643941 5473 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 8.129347ms I0626 00:04:39.643960 5473 replica.cpp:676] Persisted action at 0 I0626 00:04:39.643970 5473 replica.cpp:661] Replica learned NOP action at position 0 I0626 00:04:39.644207 5473 log.cpp:672] Writer started with ending position 0 I0626 00:04:39.644625 5471 leveldb.cpp:438] Reading position from leveldb took 9128ns I0626 00:04:39.646010 5476 registrar.cpp:346] Successfully fetched the registry (0B) I0626 00:04:39.646044 5476 registrar.cpp:422] Attempting to update the 'registry' I0626 00:04:39.647274 5471 log.cpp:680] Attempting to append 136 bytes to the log I0626 00:04:39.647337 5471 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0626 00:04:39.647687 5476 replica.cpp:508] Replica received write request for position 1 I0626 00:04:39.655206 5476 leveldb.cpp:343] Persisting action (155 bytes) to leveldb took 7.499736ms I0626 00:04:39.655225 5476 replica.cpp:676] Persisted action at 1 I0626 00:04:39.655467 5476 replica.cpp:655] Replica received learned notice for position 1 I0626 00:04:39.663534 5476 leveldb.cpp:343] Persisting action (157 bytes) to leveldb took 8.054929ms I0626 00:04:39.663554 5476 replica.cpp:676] Persisted action at 1 I0626 00:04:39.663563 5476 replica.cpp:661] Replica learned APPEND action at position 1 I0626 00:04:39.663890 5478 registrar.cpp:479] Successfully updated 'registry' I0626 00:04:39.663947 5478 registrar.cpp:372] Successfully recovered registrar I0626 00:04:39.663969 5476 log.cpp:699] Attempting to truncate the log to 1 I0626 00:04:39.664044 5478 master.cpp:980] Recovered 0 slaves from the Registry (98B) ; allowing 10mins for slaves to re-register I0626 00:04:39.664057 5476 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0626 00:04:39.664341 5476 replica.cpp:508] Replica received write request for position 2 I0626 00:04:39.664681 5450 containerizer.cpp:124] Using isolation: posix/cpu,posix/mem I0626 00:04:39.666721 5471 slave.cpp:168] Slave started on 173)@67.195.138.61:55423 I0626 00:04:39.666741 5471 credentials.hpp:35] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_MultipleFrameworks_G6ObtK/credential' I0626 00:04:39.666806 5471 slave.cpp:268] Slave using credential for: test-principal I0626 00:04:39.666936 5471 slave.cpp:281] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0626 00:04:39.667000 5471 slave.cpp:326] Slave hostname: juno.apache.org I0626 00:04:39.667009 5471 slave.cpp:327] Slave checkpoint: true I0626 00:04:39.667572 5478 state.cpp:33] Recovering state from '/tmp/SlaveRecoveryTest_0_MultipleFrameworks_G6ObtK/meta' I0626 00:04:39.667703 5475 status_update_manager.cpp:193] Recovering status update manager I0626 00:04:39.667840 5475 containerizer.cpp:287] Recovering containerizer I0626 00:04:39.668478 5471 slave.cpp:3128] Finished recovery I0626 00:04:39.668712 5471 slave.cpp:601] New master detected at master@67.195.138.61:55423 I0626 00:04:39.668738 5471 slave.cpp:677] Authenticating with master master@67.195.138.61:55423 I0626 00:04:39.668802 5471 slave.cpp:650] Detecting new master I0626 00:04:39.668861 5471 status_update_manager.cpp:167] New master detected at master@67.195.138.61:55423 I0626 00:04:39.668916 5471 authenticatee.hpp:128] Creating new client SASL connection I0626 00:04:39.669087 5471 master.cpp:3499] Authenticating slave(173)@67.195.138.61:55423 I0626 00:04:39.669203 5471 authenticator.hpp:156] Creating new server SASL connection I0626 00:04:39.669340 5471 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0626 00:04:39.669359 5471 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0626 00:04:39.669386 5471 authenticator.hpp:262] Received SASL authentication start I0626 00:04:39.669414 5471 authenticator.hpp:384] Authentication requires more steps I0626 00:04:39.669457 5471 authenticatee.hpp:265] Received SASL authentication step I0626 00:04:39.669514 5471 authenticator.hpp:290] Received SASL authentication step I0626 00:04:39.669534 5471 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'juno.apache.org' server FQDN: 'juno.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0626 00:04:39.669543 5471 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0626 00:04:39.669567 5471 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0626 00:04:39.669580 5471 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'juno.apache.org' server FQDN: 'juno.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0626 00:04:39.669589 5471 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0626 00:04:39.669594 5471 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0626 00:04:39.669606 5471 authenticator.hpp:376] Authentication success I0626 00:04:39.669641 5471 authenticatee.hpp:305] Authentication success I0626 00:04:39.669669 5471 master.cpp:3539] Successfully authenticated principal 'test-principal' at slave(173)@67.195.138.61:55423 I0626 00:04:39.669761 5450 sched.cpp:139] Version: 0.20.0 I0626 00:04:39.669764 5478 slave.cpp:734] Successfully authenticated with master master@67.195.138.61:55423 I0626 00:04:39.669826 5478 slave.cpp:972] Will retry registration in 3.190666ms if necessary I0626 00:04:39.669950 5471 master.cpp:2781] Registering slave at slave(173)@67.195.138.61:55423 (juno.apache.org) with id 20140626-000439-1032504131-55423-5450-0 I0626 00:04:39.669960 5475 sched.cpp:235] New master detected at master@67.195.138.61:55423 I0626 00:04:39.669977 5475 sched.cpp:285] Authenticating with master master@67.195.138.61:55423 I0626 00:04:39.670073 5471 registrar.cpp:422] Attempting to update the 'registry' I0626 00:04:39.670114 5475 authenticatee.hpp:128] Creating new client SASL connection I0626 00:04:39.670263 5475 master.cpp:3499] Authenticating scheduler-e66c50d2-2790-4d20-bc77-a57af0e1780b@67.195.138.61:55423 I0626 00:04:39.670361 5474 authenticator.hpp:156] Creating new server SASL connection I0626 00:04:39.670506 5475 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0626 00:04:39.670526 5475 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0626 00:04:39.670559 5475 authenticator.hpp:262] Received SASL authentication start I0626 00:04:39.670590 5475 authenticator.hpp:384] Authentication requires more steps I0626 00:04:39.670619 5475 authenticatee.hpp:265] Received SASL authentication step I0626 00:04:39.670650 5475 authenticator.hpp:290] Received SASL authentication step I0626 00:04:39.670670 5475 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'juno.apache.org' server FQDN: 'juno.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0626 00:04:39.670677 5475 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0626 00:04:39.670687 5475 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0626 00:04:39.670697 5475 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'juno.apache.org' server FQDN: 'juno.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0626 00:04:39.670706 5475 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0626 00:04:39.670712 5475 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0626 00:04:39.670723 5475 authenticator.hpp:376] Authentication success I0626 00:04:39.670749 5475 authenticatee.hpp:305] Authentication success I0626 00:04:39.670773 5475 master.cpp:3539] Successfully authenticated principal 'test-principal' at scheduler-e66c50d2-2790-4d20-bc77-a57af0e1780b@67.195.138.61:55423 I0626 00:04:39.670845 5475 sched.cpp:359] Successfully authenticated with master master@67.195.138.61:55423 I0626 00:04:39.670858 5475 sched.cpp:478] Sending registration request to master@67.195.138.61:55423 I0626 00:04:39.670899 5475 master.cpp:1241] Received registration request from scheduler-e66c50d2-2790-4d20-bc77-a57af0e1780b@67.195.138.61:55423 I0626 00:04:39.670922 5475 master.cpp:1201] Authorizing framework principal 'test-principal' to receive offers for role '*' I0626 00:04:39.671052 5475 master.cpp:1300] Registering framework 20140626-000439-1032504131-55423-5450-0000 at scheduler-e66c50d2-2790-4d20-bc77-a57af0e1780b@67.195.138.61:55423 I0626 00:04:39.671159 5474 sched.cpp:409] Framework registered with 20140626-000439-1032504131-55423-5450-0000 I0626 00:04:39.671185 5474 sched.cpp:423] Scheduler::registered took 10223ns I0626 00:04:39.671226 5474 hierarchical_allocator_process.hpp:331] Added framework 20140626-000439-1032504131-55423-5450-0000 I0626 00:04:39.671241 5474 hierarchical_allocator_process.hpp:724] No resources available to allocate! I0626 00:04:39.671247 5474 hierarchical_allocator_process.hpp:686] Performed allocation for 0 slaves in 8574ns I0626 00:04:39.671879 5476 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 7.48781ms I0626 00:04:39.671900 5476 replica.cpp:676] Persisted action at 2 I0626 00:04:39.672164 5471 replica.cpp:655] Replica received learned notice for position 2 I0626 00:04:39.674092 5472 slave.cpp:972] Will retry registration in 25.467893ms if necessary I0626 00:04:39.674108 5476 master.cpp:2769] Ignoring register slave message from slave(173)@67.195.138.61:55423 (juno.apache.org) as admission is already in progress I0626 00:04:39.680193 5471 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 8.01285ms I0626 00:04:39.680223 5471 leveldb.cpp:401] Deleting ~1 keys from leveldb took 11393ns I0626 00:04:39.680234 5471 replica.cpp:676] Persisted action at 2 I0626 00:04:39.680245 5471 replica.cpp:661] Replica learned TRUNCATE action at position 2 I0626 00:04:39.680585 5472 log.cpp:680] Attempting to append 326 bytes to the log I0626 00:04:39.680670 5477 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I0626 00:04:39.680953 5474 replica.cpp:508] Replica received write request for position 3 I0626 00:04:39.688521 5474 leveldb.cpp:343] Persisting action (345 bytes) to leveldb took 7.548316ms I0626 00:04:39.688542 5474 replica.cpp:676] Persisted action at 3 I0626 00:04:39.688750 5474 replica.cpp:655] Replica received learned notice for position 3 I0626 00:04:39.696851 5474 leveldb.cpp:343] Persisting action (347 bytes) to leveldb took 8.088289ms I0626 00:04:39.696869 5474 replica.cpp:676] Persisted action at 3 I0626 00:04:39.696878 5474 replica.cpp:661] Replica learned APPEND action at position 3 I0626 00:04:39.697268 5474 registrar.cpp:479] Successfully updated 'registry' I0626 00:04:39.697350 5474 log.cpp:699] Attempting to truncate the log to 3 I0626 00:04:39.697412 5474 master.cpp:2821] Registered slave 20140626-000439-1032504131-55423-5450-0 at slave(173)@67.195.138.61:55423 (juno.apache.org) I0626 00:04:39.697423 5474 master.cpp:3967] Adding slave 20140626-000439-1032504131-55423-5450-0 at slave(173)@67.195.138.61:55423 (juno.apache.org) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0626 00:04:39.697535 5474 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 I0626 00:04:39.697618 5474 slave.cpp:768] Registered with master master@67.195.138.61:55423; given slave ID 20140626-000439-1032504131-55423-5450-0 I0626 00:04:39.697754 5474 slave.cpp:781] Checkpointing SlaveInfo to '/tmp/SlaveRecoveryTest_0_MultipleFrameworks_G6ObtK/meta/slaves/20140626-000439-1032504131-55423-5450-0/slave.info' I0626 00:04:39.697762 5471 hierarchical_allocator_process.hpp:444] Added slave 20140626-000439-1032504131-55423-5450-0 (juno.apache.org) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] available) I0626 00:04:39.697845 5471 hierarchical_allocator_process.hpp:750] Offering cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140626-000439-1032504131-55423-5450-0 to framework 20140626-000439-1032504131-55423-5450-0000 I0626 00:04:39.697854 5474 slave.cpp:2325] Received ping from slave-observer(142)@67.195.138.61:55423 I0626 00:04:39.698040 5471 hierarchical_allocator_process.hpp:706] Performed allocation for slave 20140626-000439-1032504131-55423-5450-0 in 231333ns I0626 00:04:39.698051 5474 replica.cpp:508] Replica received write request for position 4 I0626 00:04:39.698118 5471 master.hpp:794] Adding offer 20140626-000439-1032504131-55423-5450-0 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140626-000439-1032504131-55423-5450-0 (juno.apache.org) I0626 00:04:39.698170 5471 master.cpp:3446] Sending 1 offers to framework 20140626-000439-1032504131-55423-5450-0000 I0626 00:04:39.698318 5471 sched.cpp:546] Scheduler::resourceOffers took 24371ns I0626 00:04:39.699718 5477 master.hpp:804] Removing offer 20140626-000439-1032504131-55423-5450-0 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140626-000439-1032504131-55423-5450-0 (juno.apache.org) I0626 00:04:39.699787 5477 master.cpp:2125] Processing reply for offers: [ 20140626-000439-1032504131-55423-5450-0 ] on slave 20140626-000439-1032504131-55423-5450-0 at slave(173)@67.195.138.61:55423 (juno.apache.org) for framework 20140626-000439-1032504131-55423-5450-0000 I0626 00:04:39.699812 5477 master.cpp:2211] Authorizing framework principal 'test-principal' to launch task 897522cc-4ec5-4904-aed0-00b6b8c41028 as user 'jenkins' I0626 00:04:39.700160 5477 master.hpp:766] Adding task 897522cc-4ec5-4904-aed0-00b6b8c41028 with resources cpus(*):1; mem(*):512 on slave 20140626-000439-1032504131-55423-5450-0 (juno.apache.org) I0626 00:04:39.700188 5477 master.cpp:2277] Launching task 897522cc-4ec5-4904-aed0-00b6b8c41028 of framework 20140626-000439-1032504131-55423-5450-0000 with resources cpus(*):1; mem(*):512 on slave 20140626-000439-1032504131-55423-5450-0 at slave(173)@67.195.138.61:55423 (juno.apache.org) I0626 00:04:39.700392 5471 slave.cpp:1003] Got assigned task 897522cc-4ec5-4904-aed0-00b6b8c41028 for framework 20140626-000439-1032504131-55423-5450-0000 I0626 00:04:39.700479 5477 hierarchical_allocator_process.hpp:546] Framework 20140626-000439-1032504131-55423-5450-0000 left cpus(*):1; mem(*):512; disk(*):1024; ports(*):[31000-32000] unused on slave 20140626-000439-1032504131-55423-5450-0 I0626 00:04:39.700505 5471 slave.cpp:3400] Checkpointing FrameworkInfo to '/tmp/SlaveRecoveryTest_0_MultipleFrameworks_G6ObtK/meta/slaves/20140626-000439-1032504131-55423-5450-0/frameworks/20140626-000439-1032504131-55423-5450-0000/framework.info' I0626 00:04:39.700597 5477 hierarchical_allocator_process.hpp:588] Framework 20140626-000439-1032504131-55423-5450-0000 filtered slave 20140626-000439-1032504131-55423-5450-0 for 5secs I0626 00:04:39.700686 5471 slave.cpp:3407] Checkpointing framework pid 'scheduler-e66c50d2-2790-4d20-bc77-a57af0e1780b@67.195.138.61:55423' to '... | 1 |
| 66 | MESOS-1559 | Allow jenkins build machine to dump stack traces of all threads when timeout | Many of the time, when jenkins build times out, we know that some test freezes at some place. However, most of the time, it's very hard to reproduce the deadlock on dev machines. I would be cool if we can dump the stack traces of all threads when jenkins build times out. Some command like the following: {noformat} echo thread apply all bt > tmp; gdb attach `pgrep lt-mesos-tests` < tmp {noformat} | 5 |
| 67 | MESOS-1567 | Add logging of the user uid when receiving SIGTERM. | We currently do not log the user id when receiving a SIGTERM, this makes debugging a bit difficult. It's easy to get this information through sigaction. | 1 |
| 68 | MESOS-1571 | Signal escalation timeout is not configurable | Even though the executor shutdown grace period is set to a larger interval, the signal escalation timeout will still be 3 seconds. It should either be configurable or dependent on EXECUTOR_SHUTDOWN_GRACE_PERIOD. Thoughts? | 2 |
| 69 | MESOS-1578 | Improve framework rate limiting by imposing the max number of outstanding messages per framework principal | * Rate limits config takes a configurable *capacity* for each principal. * To ensure that Master maintain the message order of a framework it's important that Master sends an FrameworkErrorMessage back to the scheduler to ask it to abort. | 5 |
| 70 | MESOS-1586 | Isolate system directories, e.g., per-container /tmp | Ideally, tasks should not write outside their sandbox (executor work directory) but pragmatically they may need to write to /tmp, /var/tmp, or some other directory. 1) We should include any such files in disk usage and quota. 2) We should make these "shared" directories private, i.e., each container has their own. 3) We should make the lifetime of any such files the same as the executor work directory. | 3 |
| 71 | MESOS-1587 | Report disk usage from MesosContainerizer | We should report disk usage for the executor work directory from MesosContainerizer and include in the ResourceStatistics protobuf. | 5 |
| 72 | MESOS-1590 | Allow LoadGeneratorFramework to read password from a file | It currently just reads the flag as the value of the password. | 1 |
| 73 | MESOS-1592 | Design inverse resource offer support | An "inverse" resource offer means that Mesos is requesting resources back from the framework, possibly within some time interval. This can be leveraged initially to provide more automated cluster maintenance, by offering schedulers the opportunity to move tasks to compensate for planned maintenance. Operators can set a time limit on how long to wait for schedulers to relocate tasks before the tasks are forcibly terminated. Inverse resource offers have many other potential uses, as it opens the opportunity for the allocator to attempt to move tasks in the cluster through the co-operation of the framework, possibly providing better over-subscription, fairness, etc. | 5 |
| 74 | MESOS-1594 | SlaveRecoveryTest/0.ReconcileKillTask is flaky | Observed this on Jenkins. {code} [ RUN ] SlaveRecoveryTest/0.ReconcileKillTask Using temporary directory '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_3zJ6DG' I0714 15:08:43.915114 27216 leveldb.cpp:176] Opened db in 474.695188ms I0714 15:08:43.933645 27216 leveldb.cpp:183] Compacted db in 18.068942ms I0714 15:08:43.934129 27216 leveldb.cpp:198] Created db iterator in 7860ns I0714 15:08:43.934439 27216 leveldb.cpp:204] Seeked to beginning of db in 2560ns I0714 15:08:43.934779 27216 leveldb.cpp:273] Iterated through 0 keys in the db in 1400ns I0714 15:08:43.935098 27216 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0714 15:08:43.936027 27238 recover.cpp:425] Starting replica recovery I0714 15:08:43.936225 27238 recover.cpp:451] Replica is in EMPTY status I0714 15:08:43.936867 27238 replica.cpp:638] Replica in EMPTY status received a broadcasted recover request I0714 15:08:43.937049 27238 recover.cpp:188] Received a recover response from a replica in EMPTY status I0714 15:08:43.937232 27238 recover.cpp:542] Updating replica status to STARTING I0714 15:08:43.945600 27235 master.cpp:288] Master 20140714-150843-16842879-55850-27216 (quantal) started on 127.0.1.1:55850 I0714 15:08:43.945643 27235 master.cpp:325] Master only allowing authenticated frameworks to register I0714 15:08:43.945651 27235 master.cpp:330] Master only allowing authenticated slaves to register I0714 15:08:43.945658 27235 credentials.hpp:36] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_3zJ6DG/credentials' I0714 15:08:43.945808 27235 master.cpp:359] Authorization enabled I0714 15:08:43.946369 27235 hierarchical_allocator_process.hpp:301] Initializing hierarchical allocator process with master : master@127.0.1.1:55850 I0714 15:08:43.946419 27235 master.cpp:122] No whitelist given. Advertising offers for all slaves I0714 15:08:43.946614 27235 master.cpp:1128] The newly elected leader is master@127.0.1.1:55850 with id 20140714-150843-16842879-55850-27216 I0714 15:08:43.946630 27235 master.cpp:1141] Elected as the leading master! I0714 15:08:43.946637 27235 master.cpp:959] Recovering from registrar I0714 15:08:43.946707 27235 registrar.cpp:313] Recovering registrar I0714 15:08:43.957895 27238 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 20.529301ms I0714 15:08:43.957978 27238 replica.cpp:320] Persisted replica status to STARTING I0714 15:08:43.958142 27238 recover.cpp:451] Replica is in STARTING status I0714 15:08:43.958664 27238 replica.cpp:638] Replica in STARTING status received a broadcasted recover request I0714 15:08:43.958762 27238 recover.cpp:188] Received a recover response from a replica in STARTING status I0714 15:08:43.958945 27238 recover.cpp:542] Updating replica status to VOTING I0714 15:08:43.975685 27238 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 16.646136ms I0714 15:08:43.976367 27238 replica.cpp:320] Persisted replica status to VOTING I0714 15:08:43.976824 27241 recover.cpp:556] Successfully joined the Paxos group I0714 15:08:43.977072 27242 recover.cpp:440] Recover process terminated I0714 15:08:43.980590 27236 log.cpp:656] Attempting to start the writer I0714 15:08:43.981385 27236 replica.cpp:474] Replica received implicit promise request with proposal 1 I0714 15:08:43.999141 27236 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 17.705787ms I0714 15:08:43.999222 27236 replica.cpp:342] Persisted promised to 1 I0714 15:08:44.004451 27240 coordinator.cpp:230] Coordinator attemping to fill missing position I0714 15:08:44.004914 27240 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I0714 15:08:44.021456 27240 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 16.499775ms I0714 15:08:44.021533 27240 replica.cpp:676] Persisted action at 0 I0714 15:08:44.022006 27240 replica.cpp:508] Replica received write request for position 0 I0714 15:08:44.022043 27240 leveldb.cpp:438] Reading position from leveldb took 21376ns I0714 15:08:44.035969 27240 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 13.885907ms I0714 15:08:44.036365 27240 replica.cpp:676] Persisted action at 0 I0714 15:08:44.040156 27238 replica.cpp:655] Replica received learned notice for position 0 I0714 15:08:44.058082 27238 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 17.860707ms I0714 15:08:44.058161 27238 replica.cpp:676] Persisted action at 0 I0714 15:08:44.058176 27238 replica.cpp:661] Replica learned NOP action at position 0 I0714 15:08:44.058526 27238 log.cpp:672] Writer started with ending position 0 I0714 15:08:44.058872 27238 leveldb.cpp:438] Reading position from leveldb took 25660ns I0714 15:08:44.060556 27238 registrar.cpp:346] Successfully fetched the registry (0B) I0714 15:08:44.060845 27238 registrar.cpp:422] Attempting to update the 'registry' I0714 15:08:44.062304 27238 log.cpp:680] Attempting to append 120 bytes to the log I0714 15:08:44.062866 27236 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0714 15:08:44.063154 27236 replica.cpp:508] Replica received write request for position 1 I0714 15:08:44.082813 27236 leveldb.cpp:343] Persisting action (137 bytes) to leveldb took 19.61683ms I0714 15:08:44.082890 27236 replica.cpp:676] Persisted action at 1 I0714 15:08:44.083256 27236 replica.cpp:655] Replica received learned notice for position 1 I0714 15:08:44.097398 27236 leveldb.cpp:343] Persisting action (139 bytes) to leveldb took 14.104796ms I0714 15:08:44.097475 27236 replica.cpp:676] Persisted action at 1 I0714 15:08:44.097488 27236 replica.cpp:661] Replica learned APPEND action at position 1 I0714 15:08:44.098569 27236 registrar.cpp:479] Successfully updated 'registry' I0714 15:08:44.098906 27240 log.cpp:699] Attempting to truncate the log to 1 I0714 15:08:44.099608 27240 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0714 15:08:44.100005 27240 replica.cpp:508] Replica received write request for position 2 I0714 15:08:44.100566 27236 registrar.cpp:372] Successfully recovered registrar I0714 15:08:44.101227 27239 master.cpp:986] Recovered 0 slaves from the Registry (84B) ; allowing 10mins for slaves to re-register I0714 15:08:44.118376 27240 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 18.329495ms I0714 15:08:44.118455 27240 replica.cpp:676] Persisted action at 2 I0714 15:08:44.122258 27242 replica.cpp:655] Replica received learned notice for position 2 I0714 15:08:44.137336 27242 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 15.023553ms I0714 15:08:44.137460 27242 leveldb.cpp:401] Deleting ~1 keys from leveldb took 55049ns I0714 15:08:44.137480 27242 replica.cpp:676] Persisted action at 2 I0714 15:08:44.137492 27242 replica.cpp:661] Replica learned TRUNCATE action at position 2 I0714 15:08:44.143729 27216 containerizer.cpp:124] Using isolation: posix/cpu,posix/mem I0714 15:08:44.145934 27242 slave.cpp:168] Slave started on 43)@127.0.1.1:55850 I0714 15:08:44.145953 27242 credentials.hpp:84] Loading credential for authentication from '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_Zl9DUt/credential' I0714 15:08:44.146040 27242 slave.cpp:266] Slave using credential for: test-principal I0714 15:08:44.146136 27242 slave.cpp:279] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0714 15:08:44.146198 27242 slave.cpp:324] Slave hostname: quantal I0714 15:08:44.146209 27242 slave.cpp:325] Slave checkpoint: true I0714 15:08:44.146708 27242 state.cpp:33] Recovering state from '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_Zl9DUt/meta' I0714 15:08:44.146824 27242 status_update_manager.cpp:193] Recovering status update manager I0714 15:08:44.146901 27242 containerizer.cpp:287] Recovering containerizer I0714 15:08:44.147228 27242 slave.cpp:3126] Finished recovery I0714 15:08:44.147531 27242 slave.cpp:599] New master detected at master@127.0.1.1:55850 I0714 15:08:44.147562 27242 slave.cpp:675] Authenticating with master master@127.0.1.1:55850 I0714 15:08:44.147614 27242 slave.cpp:648] Detecting new master I0714 15:08:44.147652 27242 status_update_manager.cpp:167] New master detected at master@127.0.1.1:55850 I0714 15:08:44.147691 27242 authenticatee.hpp:128] Creating new client SASL connection I0714 15:08:44.148533 27235 master.cpp:3507] Authenticating slave(43)@127.0.1.1:55850 I0714 15:08:44.148666 27235 authenticator.hpp:156] Creating new server SASL connection I0714 15:08:44.149054 27242 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0714 15:08:44.149447 27242 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0714 15:08:44.149917 27236 authenticator.hpp:262] Received SASL authentication start I0714 15:08:44.149974 27236 authenticator.hpp:384] Authentication requires more steps I0714 15:08:44.150208 27242 authenticatee.hpp:265] Received SASL authentication step I0714 15:08:44.150720 27239 authenticator.hpp:290] Received SASL authentication step I0714 15:08:44.150749 27239 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'quantal' server FQDN: 'quantal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0714 15:08:44.150758 27239 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0714 15:08:44.150771 27239 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0714 15:08:44.150781 27239 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'quantal' server FQDN: 'quantal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0714 15:08:44.150787 27239 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0714 15:08:44.150792 27239 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0714 15:08:44.150804 27239 authenticator.hpp:376] Authentication success I0714 15:08:44.150848 27239 master.cpp:3547] Successfully authenticated principal 'test-principal' at slave(43)@127.0.1.1:55850 I0714 15:08:44.157696 27242 authenticatee.hpp:305] Authentication success I0714 15:08:44.158855 27242 slave.cpp:732] Successfully authenticated with master master@127.0.1.1:55850 I0714 15:08:44.158936 27242 slave.cpp:970] Will retry registration in 10.352612ms if necessary I0714 15:08:44.161813 27216 sched.cpp:139] Version: 0.20.0 I0714 15:08:44.162608 27236 sched.cpp:235] New master detected at master@127.0.1.1:55850 I0714 15:08:44.162637 27236 sched.cpp:285] Authenticating with master master@127.0.1.1:55850 I0714 15:08:44.162747 27236 authenticatee.hpp:128] Creating new client SASL connection I0714 15:08:44.163506 27239 master.cpp:2789] Registering slave at slave(43)@127.0.1.1:55850 (quantal) with id 20140714-150843-16842879-55850-27216-0 I0714 15:08:44.164086 27238 registrar.cpp:422] Attempting to update the 'registry' I0714 15:08:44.165694 27238 log.cpp:680] Attempting to append 295 bytes to the log I0714 15:08:44.166231 27240 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I0714 15:08:44.166517 27240 replica.cpp:508] Replica received write request for position 3 I0714 15:08:44.167199 27239 master.cpp:3507] Authenticating scheduler-225679c4-a9fd-4119-9deb-c7712eba37e1@127.0.1.1:55850 I0714 15:08:44.167867 27241 authenticator.hpp:156] Creating new server SASL connection I0714 15:08:44.168058 27241 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0714 15:08:44.168081 27241 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0714 15:08:44.168107 27241 authenticator.hpp:262] Received SASL authentication start I0714 15:08:44.168149 27241 authenticator.hpp:384] Authentication requires more steps I0714 15:08:44.168176 27241 authenticatee.hpp:265] Received SASL authentication step I0714 15:08:44.168215 27241 authenticator.hpp:290] Received SASL authentication step I0714 15:08:44.168233 27241 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'quantal' server FQDN: 'quantal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0714 15:08:44.168793 27241 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0714 15:08:44.168820 27241 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0714 15:08:44.168834 27241 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'quantal' server FQDN: 'quantal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0714 15:08:44.168840 27241 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0714 15:08:44.168845 27241 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0714 15:08:44.168858 27241 authenticator.hpp:376] Authentication success I0714 15:08:44.168895 27241 authenticatee.hpp:305] Authentication success I0714 15:08:44.168970 27241 sched.cpp:359] Successfully authenticated with master master@127.0.1.1:55850 I0714 15:08:44.168987 27241 sched.cpp:478] Sending registration request to master@127.0.1.1:55850 I0714 15:08:44.169426 27239 master.cpp:1239] Queuing up registration request from scheduler-225679c4-a9fd-4119-9deb-c7712eba37e1@127.0.1.1:55850 because authentication is still in progress I0714 15:08:44.169958 27239 master.cpp:3547] Successfully authenticated principal 'test-principal' at scheduler-225679c4-a9fd-4119-9deb-c7712eba37e1@127.0.1.1:55850 I0714 15:08:44.170440 27241 slave.cpp:970] Will retry registration in 8.76707ms if necessary I0714 15:08:44.175359 27239 master.cpp:2777] Ignoring register slave message from slave(43)@127.0.1.1:55850 (quantal) as admission is already in progress I0714 15:08:44.175916 27239 master.cpp:1247] Received registration request from scheduler-225679c4-a9fd-4119-9deb-c7712eba37e1@127.0.1.1:55850 I0714 15:08:44.176298 27239 master.cpp:1207] Authorizing framework principal 'test-principal' to receive offers for role '*' I0714 15:08:44.176858 27239 master.cpp:1306] Registering framework 20140714-150843-16842879-55850-27216-0000 at scheduler-225679c4-a9fd-4119-9deb-c7712eba37e1@127.0.1.1:55850 I0714 15:08:44.177408 27236 sched.cpp:409] Framework registered with 20140714-150843-16842879-55850-27216-0000 I0714 15:08:44.177443 27236 sched.cpp:423] Scheduler::registered took 12527ns I0714 15:08:44.177727 27241 hierarchical_allocator_process.hpp:331] Added framework 20140714-150843-16842879-55850-27216-0000 I0714 15:08:44.177747 27241 hierarchical_allocator_process.hpp:724] No resources available to allocate! I0714 15:08:44.177753 27241 hierarchical_allocator_process.hpp:686] Performed allocation for 0 slaves in 8120ns I0714 15:08:44.179908 27241 slave.cpp:970] Will retry registration in 66.781028ms if necessary I0714 15:08:44.180007 27241 master.cpp:2777] Ignoring register slave message from slave(43)@127.0.1.1:55850 (quantal) as admission is already in progress I0714 15:08:44.183082 27240 leveldb.cpp:343] Persisting action (314 bytes) to leveldb took 16.533189ms I0714 15:08:44.183125 27240 replica.cpp:676] Persisted action at 3 I0714 15:08:44.183465 27240 replica.cpp:655] Replica received learned notice for position 3 I0714 15:08:44.203276 27240 leveldb.cpp:343] Persisting action (316 bytes) to leveldb took 19.768951ms I0714 15:08:44.203376 27240 replica.cpp:676] Persisted action at 3 I0714 15:08:44.203392 27240 replica.cpp:661] Replica learned APPEND action at position 3 I0714 15:08:44.204033 27240 registrar.cpp:479] Successfully updated 'registry' I0714 15:08:44.204138 27240 log.cpp:699] Attempting to truncate the log to 3 I0714 15:08:44.204221 27240 master.cpp:2829] Registered slave 20140714-150843-16842879-55850-27216-0 at slave(43)@127.0.1.1:55850 (quantal) I0714 15:08:44.204241 27240 master.cpp:3975] Adding slave 20140714-150843-16842879-55850-27216-0 at slave(43)@127.0.1.1:55850 (quantal) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0714 15:08:44.204387 27240 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 I0714 15:08:44.204489 27240 slave.cpp:766] Registered with master master@127.0.1.1:55850; given slave ID 20140714-150843-16842879-55850-27216-0 I0714 15:08:44.204745 27240 slave.cpp:779] Checkpointing SlaveInfo to '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_Zl9DUt/meta/slaves/20140714-150843-16842879-55850-27216-0/slave.info' I0714 15:08:44.204954 27240 hierarchical_allocator_process.hpp:444] Added slave 20140714-150843-16842879-55850-27216-0 (quantal) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] available) I0714 15:08:44.205023 27240 hierarchical_allocator_process.hpp:750] Offering cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140714-150843-16842879-55850-27216-0 to framework 20140714-150843-16842879-55850-27216-0000 I0714 15:08:44.205122 27240 hierarchical_allocator_process.hpp:706] Performed allocation for slave 20140714-150843-16842879-55850-27216-0 in 131192ns I0714 15:08:44.205189 27240 slave.cpp:2323] Received ping from slave-observer(32)@127.0.1.1:55850 I0714 15:08:44.205258 27240 master.hpp:801] Adding offer 20140714-150843-16842879-55850-27216-0 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140714-150843-16842879-55850-27216-0 (quantal) I0714 15:08:44.205303 27240 master.cpp:3454] Sending 1 offers to framework 20140714-150843-16842879-55850-27216-0000 I0714 15:08:44.205469 27240 sched.cpp:546] Scheduler::resourceOffers took 23591ns I0714 15:08:44.206351 27241 replica.cpp:508] Replica received write request for position 4 I0714 15:08:44.208353 27237 master.hpp:811] Removing offer 20140714-150843-16842879-55850-27216-0 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140714-150843-16842879-55850-27216-0 (quantal) I0714 15:08:44.208436 27237 master.cpp:2133] Processing reply for offers: [ 20140714-150843-16842879-55850-27216-0 ] on slave 20140714-150843-16842879-55850-27216-0 at slave(43)@127.0.1.1:55850 (quantal) for framework 20140714-150843-16842879-55850-27216-0000 I0714 15:08:44.208472 27237 master.cpp:2219] Authorizing framework principal 'test-principal' to launch task 4a6783aa-8d07-46e3-8399-2a5d047f0021 as user 'jenkins' I0714 15:08:44.208909 27237 master.hpp:773] Adding task 4a6783aa-8d07-46e3-8399-2a5d047f0021 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140714-150843-16842879-55850-27216-0 (quantal) I0714 15:08:44.208947 27237 master.cpp:2285] Launching task 4a6783aa-8d07-46e3-8399-2a5d047f0021 of framework 20140714-150843-16842879-55850-27216-0000 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140714-150843-16842879-55850-27216-0 at slave(43)@127.0.1.1:55850 (quantal) I0714 15:08:44.209090 27237 slave.cpp:1001] Got assigned task 4a6783aa-8d07-46e3-8399-2a5d047f0021 for framework 20140714-150843-16842879-55850-27216-0000 I0714 15:08:44.209190 27237 slave.cpp:3398] Checkpointing FrameworkInfo to '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_Zl9DUt/meta/slaves/20140714-150843-16842879-55850-27216-0/frameworks/20140714-150843-16842879-55850-27216-0000/framework.info' I0714 15:08:44.209413 27237 slave.cpp:3405] Checkpointing framework pid 'scheduler-225679c4-a9fd-4119-9deb-c7712eba37e1@127.0.1.1:55850' to '/tmp/SlaveRecoveryTest_0_ReconcileKillTask_Zl9DUt/meta/slaves/20140714-150843-16842879-55850-27216-0/frameworks/20140714-150843-16842879-55850-27216-0000/framework.pid' I0714 15:08... | 1 |
| 75 | MESOS-1605 | Cleanup stout build setup | While investigating stout build setup for making it installable, I came across some discrepancies. stout tests are included in libprocess's Makefile instead of stout Makefile. stout's 3rd party dependencies (e.g., picojson) live in libprocess's 3rdparty directory instead of living in stout's (non-existent) 3rd party directory. It would be nice to fix these issues before making stout installable. | 3 |
| 76 | MESOS-1615 | Create design document for Optimistic Offers | As a first step toward Optimistic Offers, take the description from the epic and build an implementation design doc that can be shared for comments. Note: the links to the working group notes and design doc are located in the [JIRA Epic|MESOS-1607]. | 8 |
| 77 | MESOS-1620 | Reconciliation does not send back tasks pending validation / authorization. | Per Vinod's feedback on https://reviews.apache.org/r/23542/, we do not send back TASK_STAGING for those tasks that are pending in the Master (validation / authorization still in progress). For both implicit and explicit task reconciliation, the master could reply with TASK_STAGING for these tasks, as this provides additional information to the framework. | 3 |
| 78 | MESOS-1624 | Apache Jenkins build fails due to -lsnappy is set when building leveldb | The failed build: https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Set-JAVA_HOME/2261/consoleFull {noformat:title=the log where -lsnappy is used when compiling leveldb} gzip -d -c ../../3rdparty/leveldb.tar.gz | tar xf - test ! -e ../../3rdparty/leveldb.patch || patch -d leveldb -p1 <../../3rdparty/leveldb.patch touch leveldb-stamp cd leveldb && \ make CC="gcc" CXX="g++" OPT="-g -g2 -O2 -Wno-unused-local-typedefs -std=c++11 -fPIC" make[5]: Entering directory `/home/jenkins/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Set-JAVA_HOME/build/mesos-0.20.0/_build/3rdparty/leveldb' g++ -pthread -lsnappy -shared -Wl,-soname -Wl,/home/jenkins/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Set-JAVA_HOME/build/mesos-0.20.0/_build/3rdparty/leveldb/libleveldb.so.1 -I. -I./include -fno-builtin-memcmp -pthread -DOS_LINUX -DLEVELDB_PLATFORM_POSIX -DSNAPPY -g -g2 -O2 -Wno-unused-local-typedefs -std=c++11 -fPIC -fPIC db/builder.cc db/c.cc db/db_impl.cc db/db_iter.cc db/dbformat.cc db/filename.cc db/log_reader.cc db/log_writer.cc db/memtable.cc db/repair.cc db/table_cache.cc db/version_edit.cc db/version_set.cc db/write_batch.cc table/block.cc table/block_builder.cc table/filter_block.cc table/format.cc table/iterator.cc table/merger.cc table/table.cc table/table_builder.cc table/two_level_iterator.cc util/arena.cc util/bloom.cc util/cache.cc util/coding.cc util/comparator.cc util/crc32c.cc util/env.cc util/env_posix.cc util/filter_policy.cc util/hash.cc util/histogram.cc util/logging.cc util/options.cc util/status.cc port/port_posix.cc -o libleveldb.so.1.4 ln -fs libleveldb.so.1.4 libleveldb.so ln -fs libleveldb.so.1.4 libleveldb.so.1 g++ -I. -I./include -fno-builtin-memcmp -pthread -DOS_LINUX -DLEVELDB_PLATFORM_POSIX -DSNAPPY -g -g2 -O2 -Wno-unused-local-typedefs -std=c++11 -fPIC -c db/builder.cc -o db/builder.o {noformat} {noformat:title=the error} /bin/bash ../libtool --tag=CXX --mode=link g++ -pthread -g -g2 -O2 -Wno-unused-local-typedefs -std=c++11 -o mesos-local local/mesos_local-main.o libmesos.la -lsasl2 -lcurl -lz -lrt libtool: link: g++ -pthread -g -g2 -O2 -Wno-unused-local-typedefs -std=c++11 -o .libs/mesos-local local/mesos_local-main.o ./.libs/libmesos.so -lsasl2 /usr/lib/x86_64-linux-gnu/libcurl.so -lz -lrt -pthread -Wl,-rpath -Wl,/home/jenkins/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Set-JAVA_HOME/build/mesos-0.20.0/_inst/lib ./.libs/libmesos.so: undefined reference to `snappy::RawCompress(char const*, unsigned long, char*, unsigned long*)' ./.libs/libmesos.so: undefined reference to `snappy::RawUncompress(char const*, unsigned long, char*)' ./.libs/libmesos.so: undefined reference to `snappy::GetUncompressedLength(char const*, unsigned long, unsigned long*)' ./.libs/libmesos.so: undefined reference to `snappy::MaxCompressedLength(unsigned long)' {noformat} | 1 |
| 79 | MESOS-1627 | Installed protobuf header files include wrong path to mesos header file | Playing with installed mesos headers, realized that we expect users to include the path to mesos directory (e.g., /usr/local/include/mesos) even though it is on the system path. This is because scheduler.pb.h etc include "mesos.pb.h" instead of "mesos/mesos.pb.h". | 2 |
| 80 | MESOS-1629 | GLOG Initialized twice if the Framework Scheduler also uses GLOG | {noformat} Could not create logging file: No such file or directory COULD NOT CREATE A LOGGINGFILE 20140722-205220.31450!F0722 20:52:20.494424 31450 utilities.cc:317] Check failed: !IsGoogleLoggingInitialized() You called InitGoogleLogging() twice! *** Check failure stack trace: *** @ 0x4399ce google::LogMessage::Fail() @ 0x43991d google::LogMessage::SendToLog() @ 0x43932e google::LogMessage::Flush() @ 0x43c0e5 google::LogMessageFatal::~LogMessageFatal() @ 0x44089f google::glog_internal_namespace_::InitGoogleLoggingUtilities() @ 0x43c409 google::InitGoogleLogging() @ 0x7f0bdd43b55c mesos::internal::logging::initialize() @ 0x7f0bdcf9564d mesos::scheduler::MesosProcess::MesosProcess() @ 0x7f0bdcf92de0 mesos::scheduler::Mesos::Mesos() @ 0x421483 heron::mesos::Scheduler::Scheduler() @ 0x4305dc main @ 0x7f0bd97159c4 __libc_start_main @ 0x420869 (unknown) Aborted {noformat} | 2 |
| 81 | MESOS-1645 | 0.20.0 Release | I would like to volunteer to be the release manager for 0.20.0, which will be releasing the following major features: - Docker support in Mesos (MESOS-1524) - Container level network monitoring for mesos containerizer (MESOS-1228) - Authorization (MESOS-1342) - Framework rate limiting (MESOS-1306) - Enable building against installed third-party dependencies (MESOS-1071) I would like to track blockers for the release on this ticket. | 5 |
| 82 | MESOS-1649 | Network isolator should tolerate slave crashes while doing isolate/cleanup. | A slave may crash while we are installing/removing filters. The slave recovery for the network isolator should tolerate those partially installed filters. Also, we want to avoid leaking a filter on host eth0 and host lo. The current code cannot tolerate that, thus may cause the following error: {noformat} Failed to perform recovery: Collect failed: Failed to recover container d409a100-2afb-497c-864f-fe3002cf65d9 with pid 50405: No ephemeral ports found To remedy this do as follows: Step 1: rm -f /var/lib/mesos/meta/slaves/latest This ensures slave doesn't recover old live executors. Step 2: Restart the slave. {noformat} | 3 |
| 83 | MESOS-1664 | Inform framework when rate limiting is active | When we rate-limit messages from a framework, we should let them know so they can proactively back-off to avoid putting extra pressure on the master. | 3 |
| 84 | MESOS-1666 | Set maximum executors per slave to avoid overcommit of ephemeral ports | With network isolation, we statically assign ephemeral port ranges. As such there is a upper bound on the number of containers each slave can support. We should avoid sending offers for slaves that have hit that limit as any tasks will fail to launch and will be LOST. | 1 |
| 85 | MESOS-1668 | Handle a temporary one-way master --> slave socket closure. | In MESOS-1529, we realized that it's possible for a slave to remain disconnected in the master if the following occurs: → Master and Slave connected operating normally. → Temporary one-way network failure, master→slave link breaks. → Master marks slave as disconnected. → Network restored and health checking continues normally, slave is not removed as a result. Slave does not attempt to re-register since it is receiving pings once again. → Slave remains disconnected according to the master, and the slave does not try to re-register. Bad! We were originally thinking of using a failover timeout in the master to remove these slaves that don't re-register. However, it can be dangerous when ZooKeeper issues are preventing the slave from re-registering with the master; we do not want to remove a ton of slaves in this situation. Rather, when the slave is health checking correctly but does not re-register within a timeout, we could send a registration request from the master to the slave, telling the slave that it must re-register. This message could also be used when receiving status updates (or other messages) from slaves that are disconnected in the master. | 2 |
| 86 | MESOS-1671 | Expose executor metrics for slave. | Expose the following metrics: slave/executors_registering slave/executors_running slave/executors_terminating slave/executors_terminated | 2 |
| 87 | MESOS-1672 | Add filter to allocator resourcesRecovered method | The allocator already allows filters to be added when resources are unused. It is useful to also allow the same behaviour in {{resourcesRecovered}}. | 2 |
| 88 | MESOS-1673 | The value of MASTER_PING_TIMEOUT is non-deterministic | Right now, it is declared as follows: {noformat} const Duration MASTER_PING_TIMEOUT = master::SLAVE_PING_TIMEOUT * master::MAX_SLAVE_PING_TIMEOUTS {noformat} Since static initialization order in C++ is undefined, MASTER_PING_TIMEOUT's value is non-deterministic. We've already observed that in tests (where MASTER_PING_TIMEOUT == 0). | 1 |
| 89 | MESOS-1674 | Kill private_resources and treat 'ephemeral_ports' as a resource. | As the first step to solve MESOS-1654, we need to kill private_resources in SlaveInfo and add a 'ephemeral_ports' resource. For now, the slave and the port mapping isolator will simply ignore the 'ephemeral_ports' resource in ExecutorInfo and TaskInfo, and make allocation by itself. We will revisit this once the overcommit race (MESOS-1466) is fixed. | 3 |
| 90 | MESOS-1676 | ZooKeeperMasterContenderDetectorTest.MasterDetectorTimedoutSession is flaky | {noformat:title=} [ RUN ] ZooKeeperMasterContenderDetectorTest.MasterDetectorTimedoutSession I0806 01:18:37.648684 17458 zookeeper_test_server.cpp:158] Started ZooKeeperTestServer on port 42069 2014-08-06 01:18:37,650:17458(0x2b4679ca5700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2014-08-06 01:18:37,650:17458(0x2b4679ca5700):ZOO_INFO@log_env@716: Client environment:host.name=lucid 2014-08-06 01:18:37,650:17458(0x2b4679ca5700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2014-08-06 01:18:37,650:17458(0x2b4679ca5700):ZOO_INFO@log_env@724: Client environment:os.arch=2.6.32-64-generic 2014-08-06 01:18:37,650:17458(0x2b4679ca5700):ZOO_INFO@log_env@725: Client environment:os.version=#128-Ubuntu SMP Tue Jul 15 08:32:40 UTC 2014 2014-08-06 01:18:37,651:17458(0x2b4679ca5700):ZOO_INFO@log_env@733: Client environment:user.name=(null) 2014-08-06 01:18:37,651:17458(0x2b4679ca5700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins 2014-08-06 01:18:37,651:17458(0x2b4679ca5700):ZOO_INFO@log_env@753: Client environment:user.dir=/var/jenkins/workspace/mesos-ubuntu-10.04-gcc/src 2014-08-06 01:18:37,651:17458(0x2b4679ca5700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:42069 sessionTimeout=5000 watcher=0x2b467450bc00 sessionId=0 sessionPasswd=<null> context=0x1682db0 flags=0 2014-08-06 01:18:37,656:17458(0x2b468638b700):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:42069] 2014-08-06 01:18:37,669:17458(0x2b468638b700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:42069], sessionId=0x147aa6601cf0000, negotiated timeout=6000 I0806 01:18:37.671725 17486 group.cpp:313] Group process (group(37)@127.0.1.1:55561) connected to ZooKeeper I0806 01:18:37.671758 17486 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I0806 01:18:37.671771 17486 group.cpp:385] Trying to create path '/mesos' in ZooKeeper 2014-08-06 01:18:39,101:17458(0x2b4687394700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:36197] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:42,441:17458(0x2b4687394700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:36197] zk retcode=-4, errno=111(Connection refused): server refused to accept the client I0806 01:18:42.656673 17481 contender.cpp:131] Joining the ZK group I0806 01:18:42.662484 17484 contender.cpp:247] New candidate (id='0') has entered the contest for leadership I0806 01:18:42.663754 17481 detector.cpp:138] Detected a new leader: (id='0') I0806 01:18:42.663884 17481 group.cpp:658] Trying to get '/mesos/info_0000000000' in ZooKeeper I0806 01:18:42.664788 17483 detector.cpp:426] A new leading master (UPID=@128.150.152.0:10000) is detected 2014-08-06 01:18:42,666:17458(0x2b4679ea6700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2014-08-06 01:18:42,666:17458(0x2b4679ea6700):ZOO_INFO@log_env@716: Client environment:host.name=lucid 2014-08-06 01:18:42,666:17458(0x2b4679ea6700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2014-08-06 01:18:42,666:17458(0x2b4679ea6700):ZOO_INFO@log_env@724: Client environment:os.arch=2.6.32-64-generic 2014-08-06 01:18:42,666:17458(0x2b4679ea6700):ZOO_INFO@log_env@725: Client environment:os.version=#128-Ubuntu SMP Tue Jul 15 08:32:40 UTC 2014 2014-08-06 01:18:42,666:17458(0x2b4679ea6700):ZOO_INFO@log_env@733: Client environment:user.name=(null) 2014-08-06 01:18:42,666:17458(0x2b4679ea6700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins 2014-08-06 01:18:42,666:17458(0x2b4679ea6700):ZOO_INFO@log_env@753: Client environment:user.dir=/var/jenkins/workspace/mesos-ubuntu-10.04-gcc/src 2014-08-06 01:18:42,666:17458(0x2b4679ea6700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:42069 sessionTimeout=5000 watcher=0x2b467450bc00 sessionId=0 sessionPasswd=<null> context=0x15c00f0 flags=0 2014-08-06 01:18:42,668:17458(0x2b4686d91700):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:42069] 2014-08-06 01:18:42,672:17458(0x2b4686d91700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:42069], sessionId=0x147aa6601cf0001, negotiated timeout=6000 I0806 01:18:42.673542 17485 group.cpp:313] Group process (group(38)@127.0.1.1:55561) connected to ZooKeeper I0806 01:18:42.673570 17485 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I0806 01:18:42.673580 17485 group.cpp:385] Trying to create path '/mesos' in ZooKeeper 2014-08-06 01:18:46,796:17458(0x2b468638b700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 2131ms 2014-08-06 01:18:46,796:17458(0x2b468638b700):ZOO_ERROR@handle_socket_error_msg@1643: Socket [127.0.0.1:42069] zk retcode=-7, errno=110(Connection timed out): connection to 127.0.0.1:42069 timed out (exceeded timeout by 131ms) 2014-08-06 01:18:46,796:17458(0x2b468638b700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 2131ms 2014-08-06 01:18:46,796:17458(0x2b4686d91700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 2115ms 2014-08-06 01:18:46,796:17458(0x2b4686d91700):ZOO_ERROR@handle_socket_error_msg@1643: Socket [127.0.0.1:42069] zk retcode=-7, errno=110(Connection timed out): connection to 127.0.0.1:42069 timed out (exceeded timeout by 115ms) 2014-08-06 01:18:46,796:17458(0x2b4686d91700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 2115ms 2014-08-06 01:18:46,799:17458(0x2b4687394700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 1025ms 2014-08-06 01:18:46,800:17458(0x2b4687394700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:36197] zk retcode=-4, errno=111(Connection refused): server refused to accept the client I0806 01:18:46.806895 17486 group.cpp:418] Lost connection to ZooKeeper, attempting to reconnect ... I0806 01:18:46.807857 17479 group.cpp:418] Lost connection to ZooKeeper, attempting to reconnect ... I0806 01:18:47.669064 17482 contender.cpp:131] Joining the ZK group 2014-08-06 01:18:47,669:17458(0x2b4686d91700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 2989ms 2014-08-06 01:18:47,669:17458(0x2b4686d91700):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:42069] 2014-08-06 01:18:47,671:17458(0x2b4686d91700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:42069], sessionId=0x147aa6601cf0001, negotiated timeout=6000 I0806 01:18:47.682868 17485 contender.cpp:247] New candidate (id='1') has entered the contest for leadership I0806 01:18:47.683404 17482 group.cpp:313] Group process (group(38)@127.0.1.1:55561) reconnected to ZooKeeper I0806 01:18:47.683445 17482 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I0806 01:18:47.685998 17482 detector.cpp:138] Detected a new leader: (id='0') I0806 01:18:47.686142 17482 group.cpp:658] Trying to get '/mesos/info_0000000000' in ZooKeeper I0806 01:18:47.687289 17479 detector.cpp:426] A new leading master (UPID=@128.150.152.0:10000) is detected 2014-08-06 01:18:47,687:17458(0x2b467a2a8700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2014-08-06 01:18:47,687:17458(0x2b467a2a8700):ZOO_INFO@log_env@716: Client environment:host.name=lucid 2014-08-06 01:18:47,687:17458(0x2b467a2a8700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2014-08-06 01:18:47,687:17458(0x2b467a2a8700):ZOO_INFO@log_env@724: Client environment:os.arch=2.6.32-64-generic 2014-08-06 01:18:47,687:17458(0x2b467a2a8700):ZOO_INFO@log_env@725: Client environment:os.version=#128-Ubuntu SMP Tue Jul 15 08:32:40 UTC 2014 2014-08-06 01:18:47,687:17458(0x2b467a2a8700):ZOO_INFO@log_env@733: Client environment:user.name=(null) 2014-08-06 01:18:47,687:17458(0x2b467a2a8700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins 2014-08-06 01:18:47,687:17458(0x2b467a2a8700):ZOO_INFO@log_env@753: Client environment:user.dir=/var/jenkins/workspace/mesos-ubuntu-10.04-gcc/src 2014-08-06 01:18:47,687:17458(0x2b467a2a8700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:42069 sessionTimeout=5000 watcher=0x2b467450bc00 sessionId=0 sessionPasswd=<null> context=0x2b467c0421c0 flags=0 2014-08-06 01:18:47,699:17458(0x2b4687de6700):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:42069] 2014-08-06 01:18:47,712:17458(0x2b4687de6700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:42069], sessionId=0x147aa6601cf0002, negotiated timeout=6000 I0806 01:18:47.712846 17479 group.cpp:313] Group process (group(39)@127.0.1.1:55561) connected to ZooKeeper I0806 01:18:47.712873 17479 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I0806 01:18:47.712882 17479 group.cpp:385] Trying to create path '/mesos' in ZooKeeper I0806 01:18:47.714648 17479 detector.cpp:138] Detected a new leader: (id='0') I0806 01:18:47.714759 17479 group.cpp:658] Trying to get '/mesos/info_0000000000' in ZooKeeper I0806 01:18:47.716130 17479 detector.cpp:426] A new leading master (UPID=@128.150.152.0:10000) is detected 2014-08-06 01:18:47,718:17458(0x2b4686d91700):ZOO_ERROR@handle_socket_error_msg@1721: Socket [127.0.0.1:42069] zk retcode=-4, errno=112(Host is down): failed while receiving a server response I0806 01:18:47.718889 17479 group.cpp:418] Lost connection to ZooKeeper, attempting to reconnect ... 2014-08-06 01:18:47,720:17458(0x2b4687de6700):ZOO_ERROR@handle_socket_error_msg@1721: Socket [127.0.0.1:42069] zk retcode=-4, errno=112(Host is down): failed while receiving a server response I0806 01:18:47.720788 17484 group.cpp:418] Lost connection to ZooKeeper, attempting to reconnect ... I0806 01:18:47.724663 17458 zookeeper_test_server.cpp:122] Shutdown ZooKeeperTestServer on port 42069 2014-08-06 01:18:48,798:17458(0x2b468638b700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 4133ms 2014-08-06 01:18:48,798:17458(0x2b468638b700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:49,720:17458(0x2b4686d91700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 33ms 2014-08-06 01:18:49,721:17458(0x2b4686d91700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:49,722:17458(0x2b4687de6700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:50,136:17458(0x2b4687394700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:36197] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:50,800:17458(0x2b468638b700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:51,723:17458(0x2b4686d91700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:51,723:17458(0x2b4687de6700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:52,801:17458(0x2b468638b700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client W0806 01:18:52.842553 17481 group.cpp:456] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=147aa6601cf0000) expiration I0806 01:18:52.842911 17481 group.cpp:472] ZooKeeper session expired I0806 01:18:52.843468 17485 detector.cpp:126] The current leader (id=0) is lost I0806 01:18:52.843483 17485 detector.cpp:138] Detected a new leader: None I0806 01:18:52.843618 17485 contender.cpp:196] Membership cancelled: 0 2014-08-06 01:18:52,843:17458(0x2b4679aa4700):ZOO_INFO@zookeeper_close@2522: Freeing zookeeper resources for sessionId=0x147aa6601cf0000 2014-08-06 01:18:52,844:17458(0x2b4679aa4700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2014-08-06 01:18:52,844:17458(0x2b4679aa4700):ZOO_INFO@log_env@716: Client environment:host.name=lucid 2014-08-06 01:18:52,844:17458(0x2b4679aa4700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2014-08-06 01:18:52,844:17458(0x2b4679aa4700):ZOO_INFO@log_env@724: Client environment:os.arch=2.6.32-64-generic 2014-08-06 01:18:52,844:17458(0x2b4679aa4700):ZOO_INFO@log_env@725: Client environment:os.version=#128-Ubuntu SMP Tue Jul 15 08:32:40 UTC 2014 2014-08-06 01:18:52,844:17458(0x2b4679aa4700):ZOO_INFO@log_env@733: Client environment:user.name=(null) 2014-08-06 01:18:52,844:17458(0x2b4679aa4700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins 2014-08-06 01:18:52,844:17458(0x2b4679aa4700):ZOO_INFO@log_env@753: Client environment:user.dir=/var/jenkins/workspace/mesos-ubuntu-10.04-gcc/src 2014-08-06 01:18:52,844:17458(0x2b4679aa4700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:42069 sessionTimeout=5000 watcher=0x2b467450bc00 sessionId=0 sessionPasswd=<null> context=0x1349ad0 flags=0 2014-08-06 01:18:52,844:17458(0x2b468698f700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:53,473:17458(0x2b4687394700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:36197] zk retcode=-4, errno=111(Connection refused): server refused to accept the client W0806 01:18:53.720684 17480 group.cpp:456] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=147aa6601cf0001) expiration I0806 01:18:53.721132 17480 group.cpp:472] ZooKeeper session expired I0806 01:18:53.721516 17479 detector.cpp:126] The current leader (id=0) is lost I0806 01:18:53.721534 17479 detector.cpp:138] Detected a new leader: None I0806 01:18:53.721696 17479 contender.cpp:196] Membership cancelled: 1 2014-08-06 01:18:53,721:17458(0x2b46798a3700):ZOO_INFO@zookeeper_close@2522: Freeing zookeeper resources for sessionId=0x147aa6601cf0001 2014-08-06 01:18:53,722:17458(0x2b46798a3700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2014-08-06 01:18:53,722:17458(0x2b46798a3700):ZOO_INFO@log_env@716: Client environment:host.name=lucid 2014-08-06 01:18:53,722:17458(0x2b46798a3700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2014-08-06 01:18:53,722:17458(0x2b46798a3700):ZOO_INFO@log_env@724: Client environment:os.arch=2.6.32-64-generic 2014-08-06 01:18:53,722:17458(0x2b46798a3700):ZOO_INFO@log_env@725: Client environment:os.version=#128-Ubuntu SMP Tue Jul 15 08:32:40 UTC 2014 2014-08-06 01:18:53,722:17458(0x2b46798a3700):ZOO_INFO@log_env@733: Client environment:user.name=(null) 2014-08-06 01:18:53,722:17458(0x2b46798a3700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins 2014-08-06 01:18:53,722:17458(0x2b46798a3700):ZOO_INFO@log_env@753: Client environment:user.dir=/var/jenkins/workspace/mesos-ubuntu-10.04-gcc/src 2014-08-06 01:18:53,722:17458(0x2b46798a3700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:42069 sessionTimeout=5000 watcher=0x2b467450bc00 sessionId=0 sessionPasswd=<null> context=0x16a0550 flags=0 2014-08-06 01:18:53,723:17458(0x2b4686f92700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:53,726:17458(0x2b4687de6700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client W0806 01:18:53.730258 17479 group.cpp:456] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=147aa6601cf0002) expiration I0806 01:18:53.730736 17479 group.cpp:472] ZooKeeper session expired I0806 01:18:53.731081 17481 detector.cpp:126] The current leader (id=0) is lost I0806 01:18:53.731132 17481 detector.cpp:138] Detected a new leader: None 2014-08-06 01:18:53,731:17458(0x2b46796a2700):ZOO_INFO@zookeeper_close@2522: Freeing zookeeper resources for sessionId=0x147aa6601cf0002 2014-08-06 01:18:53,731:17458(0x2b46796a2700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2014-08-06 01:18:53,731:17458(0x2b46796a2700):ZOO_INFO@log_env@716: Client environment:host.name=lucid 2014-08-06 01:18:53,731:17458(0x2b46796a2700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2014-08-06 01:18:53,731:17458(0x2b46796a2700):ZOO_INFO@log_env@724: Client environment:os.arch=2.6.32-64-generic 2014-08-06 01:18:53,731:17458(0x2b46796a2700):ZOO_INFO@log_env@725: Client environment:os.version=#128-Ubuntu SMP Tue Jul 15 08:32:40 UTC 2014 2014-08-06 01:18:53,731:17458(0x2b46796a2700):ZOO_INFO@log_env@733: Client environment:user.name=(null) 2014-08-06 01:18:53,731:17458(0x2b46796a2700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins 2014-08-06 01:18:53,732:17458(0x2b46796a2700):ZOO_INFO@log_env@753: Client environment:user.dir=/var/jenkins/workspace/mesos-ubuntu-10.04-gcc/src 2014-08-06 01:18:53,732:17458(0x2b46796a2700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:42069 sessionTimeout=5000 watcher=0x2b467450bc00 sessionId=0 sessionPasswd=<null> context=0x2b467c035f30 flags=0 2014-08-06 01:18:53,733:17458(0x2b4687be5700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:54,512:17458(0x2b468698f700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:55,393:17458(0x2b4686f92700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:55,403:17458(0x2b4687be5700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:56,301:17458(0x2b468698f700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 122ms 2014-08-06 01:18:56,302:17458(0x2b468698f700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:56,809:17458(0x2b4687394700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:36197] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:57,939:17458(0x2b4686f92700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 879ms 2014-08-06 01:18:57,940:17458(0x2b4686f92700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client 2014-08-06 01:18:57,940:17458(0x2b4687be5700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 870ms 2014-08-06 01:18:57,940:17458(0x2b4687be5700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:42069] zk retcode=-4, errno=111(Connection refused): server refused to accept the client tests/master_contender_detector_tests.cpp:574: Failure Failed to wait 10secs for leaderReconnecting 2014-08-06 01:18:57,941:17458(0x2b46794a0120):ZOO_INFO@zookeeper_close@2522: Freeing zookeeper resources for sessionId=0 I0806 01:18:57.949972 17458 contender.cpp:186] Now cancelling the membership: 1 2014-08-06 01:1... | 1 |
| 91 | MESOS-1677 | AllocatorTest.FrameworkReregistersFirst is flaky. | {noformat} GMOCK WARNING: Uninteresting mock function call - taking default action specified at: ../../../mesos/src/tests/mesos.hpp:566: Function call: resourcesRecovered(@0x7f38f40043e8 20140806-190304-2081170186-36159-24511-0000, @0x7f38f40043c8 20140806-190304-2081170186-36159-24511-0, @0x7f38f40043b0 { cpus(*):2, mem(*):1024, disk(*):464204, ports(*):[31000-32000] }) {noformat} | 2 |
| 92 | MESOS-1683 | Create user doc for framework rate limiting feature | Create a Markdown doc under /docs | 2 |
| 93 | MESOS-1690 | Expose metric for container destroy failures | Increment counter when container destroy fails. | 3 |
| 94 | MESOS-1694 | Future::failure should return a const string& | NULL | 1 |
| 95 | MESOS-1695 | The stats.json endpoint on the slave exposes "registered" as a string. | The slave is currently exposing a string value for the "registered" statistic, this should be a number: {code} slave:5051/stats.json { "recovery_errors": 0, "registered": "1", "slave/executors_registering": 0, ... } {code} Should be a pretty straightforward fix, looks like this first originated back in 2013: {code} commit b8291304e1523eb67ea8dc5f195cdb0d8e7d8348 Author: Vinod Kone <vinod@twitter.com> Date: Wed Jul 3 12:37:36 2013 -0700 Added a "registered" key/value pair to slave's stats.json. Review: https://reviews.apache.org/r/12256 diff --git a/src/slave/http.cpp b/src/slave/http.cpp index dc2955f..dd51516 100644 --- a/src/slave/http.cpp +++ b/src/slave/http.cpp @@ -281,6 +281,8 @@ Future<Response> Slave::Http::stats(const Request& request) object.values["lost_tasks"] = slave.stats.tasks[TASK_LOST]; object.values["valid_status_updates"] = slave.stats.validStatusUpdates; object.values["invalid_status_updates"] = slave.stats.invalidStatusUpdates; + object.values["registered"] = slave.master ? "1" : "0"; + return OK(object, request.query.get("jsonp")); } {code} | 1 |
| 96 | MESOS-1696 | Improve reconciliation between master and slave. | As we update the Master to keep tasks in memory until they are both terminal and acknowledged (MESOS-1410), the lifetime of tasks in Mesos will look as follows: {code} Master Slave {} {} {Tn} {} // Master receives Task T, non-terminal. Forwards to slave. {Tn} {Tn} // Slave receives Task T, non-terminal. {Tn} {Tt} // Task becomes terminal on slave. Update forwarded. {Tt} {Tt} // Master receives update, forwards to framework. {} {Tt} // Master receives ack, forwards to slave. {} {} // Slave receives ack. {code} In the current form of reconciliation, the slave sends to the master all tasks that are not both terminal and acknowledged. At any point in the above lifecycle, the slave's re-registration message can reach the master. Note the following properties: *(1)* The master may have a non-terminal task, not present in the slave's re-registration message. *(2)* The master may have a non-terminal task, present in the slave's re-registration message but in a different state. *(3)* The slave's re-registration message may contain a terminal unacknowledged task unknown to the master. In the current master / slave [reconciliation|https://github.com/apache/mesos/blob/0.19.1/src/master/master.cpp#L3146] code, the master assumes that case (1) is because a launch task message was dropped, and it sends TASK_LOST. We've seen above that (1) can happen even when the task reaches the slave correctly, so this can lead to inconsistency! After chatting with [~vinodkone], we're considering updating the reconciliation to occur as follows: → Slave sends all tasks that are not both terminal and acknowledged, during re-registration. This is the same as before. → If the master sees tasks that are missing in the slave, the master sends the tasks that need to be reconciled to the slave for the tasks. This can be piggy-backed on the re-registration message. → The slave will send TASK_LOST if the task is not known to it. Preferably in a retried manner, unless we update socket closure on the slave to force a re-registration. | 3 |
| 97 | MESOS-1698 | make check segfaults | Observed this on Apache CI: https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Set-JAVA_HOME/2331/consoleFull It looks like the segfault happens before any tests are run. So I suspect somewhere in the setup phase of the tests. {code} mv -f .deps/tests-time_tests.Tpo .deps/tests-time_tests.Po /bin/bash ./libtool --tag=CXX --mode=link g++ -g -g2 -O2 -Wno-unused-local-typedefs -std=c++11 -o tests tests-decoder_tests.o tests-encoder_tests.o tests-http_tests.o tests-io_tests.o tests-main.o tests-mutex_tests.o tests-metrics_tests.o tests-owned_tests.o tests-process_tests.o tests-queue_tests.o tests-reap_tests.o tests-sequence_tests.o tests-shared_tests.o tests-statistics_tests.o tests-subprocess_tests.o tests-system_tests.o tests-timeseries_tests.o tests-time_tests.o 3rdparty/libgmock.la libprocess.la 3rdparty/glog-0.3.3/libglog.la 3rdparty/libry_http_parser.la 3rdparty/libev-4.15/libev.la -lz -lrt libtool: link: g++ -g -g2 -O2 -Wno-unused-local-typedefs -std=c++11 -o tests tests-decoder_tests.o tests-encoder_tests.o tests-http_tests.o tests-io_tests.o tests-main.o tests-mutex_tests.o tests-metrics_tests.o tests-owned_tests.o tests-process_tests.o tests-queue_tests.o tests-reap_tests.o tests-sequence_tests.o tests-shared_tests.o tests-statistics_tests.o tests-subprocess_tests.o tests-system_tests.o tests-timeseries_tests.o tests-time_tests.o 3rdparty/.libs/libgmock.a ./.libs/libprocess.a /home/jenkins/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Set-JAVA_HOME/build/3rdparty/libprocess/3rdparty/glog-0.3.3/.libs/libglog.a /home/jenkins/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Set-JAVA_HOME/build/3rdparty/libprocess/3rdparty/libev-4.15/.libs/libev.a 3rdparty/glog-0.3.3/.libs/libglog.a -lpthread 3rdparty/.libs/libry_http_parser.a 3rdparty/libev-4.15/.libs/libev.a -lm -lz -lrt make[5]: Leaving directory `/home/jenkins/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Set-JAVA_HOME/build/3rdparty/libprocess' make check-local make[5]: Entering directory `/home/jenkins/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Set-JAVA_HOME/build/3rdparty/libprocess' ./tests Note: Google Test filter = [==========] Running 0 tests from 0 test cases. [==========] 0 tests from 0 test cases ran. (0 ms total) [ PASSED ] 0 tests. YOU HAVE 3 DISABLED TESTS make[5]: *** [check-local] Segmentation fault make[5]: Leaving directory `/home/jenkins/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Set-JAVA_HOME/build/3rdparty/libprocess' make[4]: *** [check-am] Error 2 make[4]: Leaving directory `/home/jenkins/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Set-JAVA_HOME/build/3rdparty/libprocess' make[3]: *** [check-recursive] Error 1 make[3]: Leaving directory `/home/jenkins/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Set-JAVA_HOME/build/3rdparty/libprocess' make[2]: *** [check-recursive] Error 1 make[2]: Leaving directory `/home/jenkins/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Set-JAVA_HOME/build/3rdparty' make[1]: *** [check] Error 2 make[1]: Leaving directory `/home/jenkins/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Set-JAVA_HOME/build/3rdparty' make: *** [check-recursive] Error 1 Build step 'Execute shell' marked build as failure Sending e-mails to: dev@mesos.apache.org benjamin.hindman@gmail.com dhamon@twitter.com yujie.jay@gmail.com Finished: FAILURE {code} | 2 |
| 98 | MESOS-1702 | Add document for network monitoring. | The doc should tell the user how to use the new network monitoring feature. | 2 |
| 99 | MESOS-1703 | better error message when replicated log hasn't been initialized | Aurora uses the mesos replicated log. If you don't run "mesos-log initialize" before starting aurora you'll get INFO messages in your aurora log: {code} I0814 15:18:38.346638 25141 replica.cpp:633] Replica in EMPTY status received a broadcasted recover request I0814 15:18:38.346796 25132 recover.cpp:220] Received a recover response from a replica in EMPTY status {code} It is has been deemed too dangerous to automatically run mesos-log initialize for the user (see AURORA-243). It would be helpful if that error message was made more friendly and at the ERROR level. The message could explain what the user should do and the implications of doing so. Links to the docs would be helpful. See http://wilderness.apache.org/channels/?f=aurora/2014-08-14#1408055261 for context | 1 |
| 100 | MESOS-1705 | SubprocessTest.Status sometimes flakes out | It's a pretty rare event, but happened more then once. [ RUN ] SubprocessTest.Status *** Aborted at 1408023909 (unix time) try "date -d @1408023909" if you are using GNU date *** PC: @ 0x35700094b1 (unknown) *** SIGTERM (@0x3e8000041d8) received by PID 16872 (TID 0x7fa9ea426780) from PID 16856; stack trace: *** @ 0x3570435cb0 (unknown) @ 0x35700094b1 (unknown) @ 0x3570009d9f (unknown) @ 0x357000e726 (unknown) @ 0x3570015185 (unknown) @ 0x5ead42 process::childMain() @ 0x5ece8d std::_Function_handler<>::_M_invoke() @ 0x5eac9c process::defaultClone() @ 0x5ebbd4 process::subprocess() @ 0x55a229 process::subprocess() @ 0x55a846 process::subprocess() @ 0x54224c SubprocessTest_Status_Test::TestBody() @ 0x7fa9ea460323 (unknown) @ 0x7fa9ea455b67 (unknown) @ 0x7fa9ea455c0e (unknown) @ 0x7fa9ea455d15 (unknown) @ 0x7fa9ea4593a8 (unknown) @ 0x7fa9ea459647 (unknown) @ 0x422466 main @ 0x3570421d65 (unknown) @ 0x4260bd (unknown) [ OK ] SubprocessTest.Status (153 ms) | 2 |
| 101 | MESOS-1712 | Automate disallowing of commits mixing mesos/libprocess/stout | For various reasons, we don't want to mix mesos/libprocess/stout changes into a single commit. Typically, it is up to the reviewee/reviewer to catch this. It wold be nice to automate this via the pre-commit hook . | 2 |
| 102 | MESOS-1715 | The slave does not send pending tasks during re-registration. | In what looks like an oversight, the pending tasks and executors in the slave (Framework::pending) are not sent in the re-registration message. For tasks, this can lead to spurious TASK_LOST notifications being generated by the master when it falsely thinks the tasks are not present on the slave. | 3 |
| 103 | MESOS-1717 | The slave does not show pending tasks in the JSON endpoints. | The slave does not show pending tasks in the /state.json endpoint. This is a bit tricky to add since we rely on knowing the executor directory. | 1 |
| 104 | MESOS-1718 | Command executor can overcommit the slave. | Currently we give a small amount of resources to the command executor, in addition to resources used by the command task: https://github.com/apache/mesos/blob/0.20.0-rc1/src/slave/slave.cpp#L2448 {code: title=} ExecutorInfo Slave::getExecutorInfo( const FrameworkID& frameworkId, const TaskInfo& task) { ... // Add an allowance for the command executor. This does lead to a // small overcommit of resources. executor.mutable_resources()->MergeFrom( Resources::parse( "cpus:" + stringify(DEFAULT_EXECUTOR_CPUS) + ";" + "mem:" + stringify(DEFAULT_EXECUTOR_MEM.megabytes())).get()); ... } {code} This leads to an overcommit of the slave. Ideally, for command tasks we can "transfer" all of the task resources to the executor at the slave / isolation level. | 3 |
| 105 | MESOS-1727 | Configure fails with ../configure: line 18439: syntax error near unexpected token `PROTOBUFPREFIX,' | I followed the "Getting started" documentation and did: {noformat} $ git clone http://git-wip-us.apache.org/repos/asf/mesos.git; cd mesos $ ./bootstrap $ mkdir build; cd build $ ../configure {noformat} which aborts with {noformat} .... .... checking whether we are using the GNU C compiler... (cached) yes checking whether gcc accepts -g... (cached) yes checking for gcc option to accept ISO C89... (cached) none needed checking dependency style of gcc... (cached) gcc3 ../configure: line 18439: syntax error near unexpected token `PROTOBUFPREFIX,' ../configure: line 18439: ` PKG_CHECK_MODULES(PROTOBUFPREFIX,' {noformat} | 2 |
| 106 | MESOS-1728 | Libprocess: report bind parameters on failure | When you attempt to start slave or master and there's another one already running there, it is nice to report what are the actual parameters to {{bind}} call that failed. | 1 |
| 107 | MESOS-1733 | Change the stout path utility to declare a single, variadic 'join' function instead of several separate declarations of various discrete arities | NULL | 5 |
| 108 | MESOS-1739 | Allow slave reconfiguration on restart | Make it so that either via a slave restart or a out of process "reconfigure" ping, the attributes and resources of a slave can be updated to be a superset of what they used to be. | 3 |
| 109 | MESOS-1748 | MasterZooKeeperTest.LostZooKeeperCluster is flaky | {noformat:title=} tests/master_tests.cpp:1795: Failure Failed to wait 10secs for slaveRegisteredMessage {noformat} Should have placed the FUTURE_MESSAGE that attempts to capture this messages before the slave starts... | 1 |
| 110 | MESOS-1749 | SlaveRecoveryTest.ShutdownSlave is flaky | {noformat} [ RUN ] SlaveRecoveryTest/0.ShutdownSlave Using temporary directory '/tmp/SlaveRecoveryTest_0_ShutdownSlave_3O5epS' I0828 21:21:46.206990 27625 leveldb.cpp:176] Opened db in 24.461837ms I0828 21:21:46.213706 27625 leveldb.cpp:183] Compacted db in 6.021499ms I0828 21:21:46.214047 27625 leveldb.cpp:198] Created db iterator in 5566ns I0828 21:21:46.214313 27625 leveldb.cpp:204] Seeked to beginning of db in 1433ns I0828 21:21:46.214515 27625 leveldb.cpp:273] Iterated through 0 keys in the db in 723ns I0828 21:21:46.214826 27625 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0828 21:21:46.215409 27642 recover.cpp:425] Starting replica recovery I0828 21:21:46.215718 27642 recover.cpp:451] Replica is in EMPTY status I0828 21:21:46.216264 27642 replica.cpp:638] Replica in EMPTY status received a broadcasted recover request I0828 21:21:46.216557 27642 recover.cpp:188] Received a recover response from a replica in EMPTY status I0828 21:21:46.216917 27642 recover.cpp:542] Updating replica status to STARTING I0828 21:21:46.221271 27645 master.cpp:286] Master 20140828-212146-16842879-45613-27625 (saucy) started on 127.0.1.1:45613 I0828 21:21:46.221812 27645 master.cpp:332] Master only allowing authenticated frameworks to register I0828 21:21:46.222038 27645 master.cpp:337] Master only allowing authenticated slaves to register I0828 21:21:46.222250 27645 credentials.hpp:36] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_ShutdownSlave_3O5epS/credentials' I0828 21:21:46.222585 27645 master.cpp:366] Authorization enabled I0828 21:21:46.222885 27642 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 5.596969ms I0828 21:21:46.223085 27642 replica.cpp:320] Persisted replica status to STARTING I0828 21:21:46.223424 27642 recover.cpp:451] Replica is in STARTING status I0828 21:21:46.223933 27642 replica.cpp:638] Replica in STARTING status received a broadcasted recover request I0828 21:21:46.224984 27642 recover.cpp:188] Received a recover response from a replica in STARTING status I0828 21:21:46.225385 27642 recover.cpp:542] Updating replica status to VOTING I0828 21:21:46.224750 27646 master.cpp:1205] The newly elected leader is master@127.0.1.1:45613 with id 20140828-212146-16842879-45613-27625 I0828 21:21:46.226132 27646 master.cpp:1218] Elected as the leading master! I0828 21:21:46.226349 27646 master.cpp:1036] Recovering from registrar I0828 21:21:46.226637 27646 registrar.cpp:313] Recovering registrar I0828 21:21:46.224473 27641 master.cpp:120] No whitelist given. Advertising offers for all slaves I0828 21:21:46.224431 27645 hierarchical_allocator_process.hpp:299] Initializing hierarchical allocator process with master : master@127.0.1.1:45613 I0828 21:21:46.240932 27642 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 15.182422ms I0828 21:21:46.241453 27642 replica.cpp:320] Persisted replica status to VOTING I0828 21:21:46.241926 27643 recover.cpp:556] Successfully joined the Paxos group I0828 21:21:46.242228 27642 recover.cpp:440] Recover process terminated I0828 21:21:46.242501 27645 log.cpp:656] Attempting to start the writer I0828 21:21:46.243247 27645 replica.cpp:474] Replica received implicit promise request with proposal 1 I0828 21:21:46.253456 27645 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 9.95472ms I0828 21:21:46.253955 27645 replica.cpp:342] Persisted promised to 1 I0828 21:21:46.254518 27645 coordinator.cpp:230] Coordinator attemping to fill missing position I0828 21:21:46.255234 27641 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I0828 21:21:46.263128 27641 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 7.484042ms I0828 21:21:46.263536 27641 replica.cpp:676] Persisted action at 0 I0828 21:21:46.263806 27641 replica.cpp:508] Replica received write request for position 0 I0828 21:21:46.263834 27641 leveldb.cpp:438] Reading position from leveldb took 14063ns I0828 21:21:46.276149 27641 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 12.295476ms I0828 21:21:46.276178 27641 replica.cpp:676] Persisted action at 0 I0828 21:21:46.276319 27641 replica.cpp:655] Replica received learned notice for position 0 I0828 21:21:46.285523 27641 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 9.185244ms I0828 21:21:46.285552 27641 replica.cpp:676] Persisted action at 0 I0828 21:21:46.285560 27641 replica.cpp:661] Replica learned NOP action at position 0 I0828 21:21:46.289685 27642 log.cpp:672] Writer started with ending position 0 I0828 21:21:46.290166 27642 leveldb.cpp:438] Reading position from leveldb took 14463ns I0828 21:21:46.297260 27642 registrar.cpp:346] Successfully fetched the registry (0B) I0828 21:21:46.297622 27642 registrar.cpp:422] Attempting to update the 'registry' I0828 21:21:46.298893 27645 log.cpp:680] Attempting to append 118 bytes to the log I0828 21:21:46.299190 27645 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0828 21:21:46.299643 27645 replica.cpp:508] Replica received write request for position 1 I0828 21:21:46.310351 27645 leveldb.cpp:343] Persisting action (135 bytes) to leveldb took 10.349409ms I0828 21:21:46.310577 27645 replica.cpp:676] Persisted action at 1 I0828 21:21:46.311039 27645 replica.cpp:655] Replica received learned notice for position 1 I0828 21:21:46.322127 27645 leveldb.cpp:343] Persisting action (137 bytes) to leveldb took 10.858061ms I0828 21:21:46.322614 27645 replica.cpp:676] Persisted action at 1 I0828 21:21:46.322875 27645 replica.cpp:661] Replica learned APPEND action at position 1 I0828 21:21:46.323480 27645 registrar.cpp:479] Successfully updated 'registry' I0828 21:21:46.323874 27645 registrar.cpp:372] Successfully recovered registrar I0828 21:21:46.323649 27639 log.cpp:699] Attempting to truncate the log to 1 I0828 21:21:46.324465 27644 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0828 21:21:46.324988 27644 replica.cpp:508] Replica received write request for position 2 I0828 21:21:46.325335 27643 master.cpp:1063] Recovered 0 slaves from the Registry (82B) ; allowing 10mins for slaves to re-register I0828 21:21:46.335847 27644 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 10.651398ms I0828 21:21:46.336320 27644 replica.cpp:676] Persisted action at 2 I0828 21:21:46.336896 27644 replica.cpp:655] Replica received learned notice for position 2 I0828 21:21:46.345854 27644 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 8.540555ms I0828 21:21:46.346261 27644 leveldb.cpp:401] Deleting ~1 keys from leveldb took 30183ns I0828 21:21:46.346282 27644 replica.cpp:676] Persisted action at 2 I0828 21:21:46.346315 27644 replica.cpp:661] Replica learned TRUNCATE action at position 2 I0828 21:21:46.356840 27625 containerizer.cpp:89] Using isolation: posix/cpu,posix/mem I0828 21:21:46.361413 27644 slave.cpp:167] Slave started on 48)@127.0.1.1:45613 I0828 21:21:46.361753 27644 credentials.hpp:84] Loading credential for authentication from '/tmp/SlaveRecoveryTest_0_ShutdownSlave_umhraW/credential' I0828 21:21:46.362046 27644 slave.cpp:274] Slave using credential for: test-principal I0828 21:21:46.362810 27644 slave.cpp:287] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0828 21:21:46.363088 27644 slave.cpp:315] Slave hostname: saucy I0828 21:21:46.363301 27644 slave.cpp:316] Slave checkpoint: true I0828 21:21:46.363986 27644 state.cpp:33] Recovering state from '/tmp/SlaveRecoveryTest_0_ShutdownSlave_umhraW/meta' I0828 21:21:46.364308 27644 status_update_manager.cpp:193] Recovering status update manager I0828 21:21:46.364600 27644 containerizer.cpp:252] Recovering containerizer I0828 21:21:46.365325 27646 slave.cpp:3204] Finished recovery I0828 21:21:46.365839 27646 slave.cpp:598] New master detected at master@127.0.1.1:45613 I0828 21:21:46.366041 27646 slave.cpp:672] Authenticating with master master@127.0.1.1:45613 I0828 21:21:46.366317 27646 slave.cpp:645] Detecting new master I0828 21:21:46.366569 27646 status_update_manager.cpp:167] New master detected at master@127.0.1.1:45613 I0828 21:21:46.366827 27646 authenticatee.hpp:128] Creating new client SASL connection I0828 21:21:46.367204 27646 master.cpp:3637] Authenticating slave(48)@127.0.1.1:45613 I0828 21:21:46.367553 27646 authenticator.hpp:156] Creating new server SASL connection I0828 21:21:46.367857 27646 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0828 21:21:46.368031 27646 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0828 21:21:46.368228 27646 authenticator.hpp:262] Received SASL authentication start I0828 21:21:46.368444 27646 authenticator.hpp:384] Authentication requires more steps I0828 21:21:46.368648 27646 authenticatee.hpp:265] Received SASL authentication step I0828 21:21:46.368924 27646 authenticator.hpp:290] Received SASL authentication step I0828 21:21:46.369120 27646 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'saucy' server FQDN: 'saucy' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0828 21:21:46.369350 27646 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0828 21:21:46.369544 27646 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0828 21:21:46.369730 27646 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'saucy' server FQDN: 'saucy' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0828 21:21:46.369958 27646 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0828 21:21:46.370131 27646 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0828 21:21:46.370311 27646 authenticator.hpp:376] Authentication success I0828 21:21:46.370518 27646 authenticatee.hpp:305] Authentication success I0828 21:21:46.370637 27642 master.cpp:3677] Successfully authenticated principal 'test-principal' at slave(48)@127.0.1.1:45613 I0828 21:21:46.371772 27641 slave.cpp:729] Successfully authenticated with master master@127.0.1.1:45613 I0828 21:21:46.371984 27641 slave.cpp:980] Will retry registration in 15.311045ms if necessary I0828 21:21:46.372643 27641 master.cpp:2836] Registering slave at slave(48)@127.0.1.1:45613 (saucy) with id 20140828-212146-16842879-45613-27625-0 I0828 21:21:46.373016 27641 registrar.cpp:422] Attempting to update the 'registry' I0828 21:21:46.374539 27641 log.cpp:680] Attempting to append 289 bytes to the log I0828 21:21:46.374876 27641 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I0828 21:21:46.375296 27641 replica.cpp:508] Replica received write request for position 3 I0828 21:21:46.376046 27625 sched.cpp:137] Version: 0.21.0 I0828 21:21:46.376374 27646 sched.cpp:233] New master detected at master@127.0.1.1:45613 I0828 21:21:46.376595 27646 sched.cpp:283] Authenticating with master master@127.0.1.1:45613 I0828 21:21:46.376857 27646 authenticatee.hpp:128] Creating new client SASL connection I0828 21:21:46.377234 27646 master.cpp:3637] Authenticating scheduler-cb5a0264-23cc-45d0-bc4c-a92fa5308158@127.0.1.1:45613 I0828 21:21:46.377496 27646 authenticator.hpp:156] Creating new server SASL connection I0828 21:21:46.377771 27646 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0828 21:21:46.377961 27646 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0828 21:21:46.378170 27646 authenticator.hpp:262] Received SASL authentication start I0828 21:21:46.378360 27646 authenticator.hpp:384] Authentication requires more steps I0828 21:21:46.378588 27639 authenticatee.hpp:265] Received SASL authentication step I0828 21:21:46.378789 27646 authenticator.hpp:290] Received SASL authentication step I0828 21:21:46.378942 27646 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'saucy' server FQDN: 'saucy' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0828 21:21:46.379091 27646 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0828 21:21:46.379298 27646 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0828 21:21:46.379539 27646 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'saucy' server FQDN: 'saucy' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0828 21:21:46.379720 27646 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0828 21:21:46.379935 27646 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0828 21:21:46.380089 27646 authenticator.hpp:376] Authentication success I0828 21:21:46.380306 27642 authenticatee.hpp:305] Authentication success I0828 21:21:46.382625 27642 sched.cpp:357] Successfully authenticated with master master@127.0.1.1:45613 I0828 21:21:46.383031 27642 sched.cpp:476] Sending registration request to master@127.0.1.1:45613 I0828 21:21:46.382928 27640 master.cpp:3677] Successfully authenticated principal 'test-principal' at scheduler-cb5a0264-23cc-45d0-bc4c-a92fa5308158@127.0.1.1:45613 I0828 21:21:46.383651 27640 master.cpp:1324] Received registration request from scheduler-cb5a0264-23cc-45d0-bc4c-a92fa5308158@127.0.1.1:45613 I0828 21:21:46.383846 27640 master.cpp:1284] Authorizing framework principal 'test-principal' to receive offers for role '*' I0828 21:21:46.384184 27640 master.cpp:1383] Registering framework 20140828-212146-16842879-45613-27625-0000 at scheduler-cb5a0264-23cc-45d0-bc4c-a92fa5308158@127.0.1.1:45613 I0828 21:21:46.384464 27640 sched.cpp:407] Framework registered with 20140828-212146-16842879-45613-27625-0000 I0828 21:21:46.384764 27640 sched.cpp:421] Scheduler::registered took 18266ns I0828 21:21:46.384600 27644 hierarchical_allocator_process.hpp:329] Added framework 20140828-212146-16842879-45613-27625-0000 I0828 21:21:46.385171 27644 hierarchical_allocator_process.hpp:691] No resources available to allocate! I0828 21:21:46.385330 27644 hierarchical_allocator_process.hpp:653] Performed allocation for 0 slaves in 160171ns I0828 21:21:46.386292 27641 leveldb.cpp:343] Persisting action (308 bytes) to leveldb took 10.815384ms I0828 21:21:46.386492 27641 replica.cpp:676] Persisted action at 3 I0828 21:21:46.386844 27641 replica.cpp:655] Replica received learned notice for position 3 I0828 21:21:46.387980 27643 slave.cpp:980] Will retry registration in 19.851524ms if necessary I0828 21:21:46.388140 27639 master.cpp:2824] Ignoring register slave message from slave(48)@127.0.1.1:45613 (saucy) as admission is already in progress I0828 21:21:46.396355 27641 leveldb.cpp:343] Persisting action (310 bytes) to leveldb took 9.275034ms I0828 21:21:46.396641 27641 replica.cpp:676] Persisted action at 3 I0828 21:21:46.396837 27641 replica.cpp:661] Replica learned APPEND action at position 3 I0828 21:21:46.397405 27641 registrar.cpp:479] Successfully updated 'registry' I0828 21:21:46.397528 27645 log.cpp:699] Attempting to truncate the log to 3 I0828 21:21:46.397878 27645 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 I0828 21:21:46.398239 27645 replica.cpp:508] Replica received write request for position 4 I0828 21:21:46.398597 27641 master.cpp:2876] Registered slave 20140828-212146-16842879-45613-27625-0 at slave(48)@127.0.1.1:45613 (saucy) I0828 21:21:46.398870 27641 master.cpp:4110] Adding slave 20140828-212146-16842879-45613-27625-0 at slave(48)@127.0.1.1:45613 (saucy) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0828 21:21:46.399178 27639 slave.cpp:763] Registered with master master@127.0.1.1:45613; given slave ID 20140828-212146-16842879-45613-27625-0 I0828 21:21:46.399521 27639 slave.cpp:776] Checkpointing SlaveInfo to '/tmp/SlaveRecoveryTest_0_ShutdownSlave_umhraW/meta/slaves/20140828-212146-16842879-45613-27625-0/slave.info' I0828 21:21:46.399961 27641 hierarchical_allocator_process.hpp:442] Added slave 20140828-212146-16842879-45613-27625-0 (saucy) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] available) I0828 21:21:46.400316 27641 hierarchical_allocator_process.hpp:728] Offering cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140828-212146-16842879-45613-27625-0 to framework 20140828-212146-16842879-45613-27625-0000 I0828 21:21:46.400158 27644 slave.cpp:2333] Received ping from slave-observer(45)@127.0.1.1:45613 I0828 21:21:46.400872 27639 master.hpp:857] Adding offer 20140828-212146-16842879-45613-27625-0 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140828-212146-16842879-45613-27625-0 (saucy) I0828 21:21:46.401105 27639 master.cpp:3584] Sending 1 offers to framework 20140828-212146-16842879-45613-27625-0000 I0828 21:21:46.401448 27639 sched.cpp:544] Scheduler::resourceOffers took 19056ns I0828 21:21:46.401700 27641 hierarchical_allocator_process.hpp:673] Performed allocation for slave 20140828-212146-16842879-45613-27625-0 in 1.430159ms I0828 21:21:46.403659 27644 master.hpp:867] Removing offer 20140828-212146-16842879-45613-27625-0 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140828-212146-16842879-45613-27625-0 (saucy) I0828 21:21:46.403903 27644 master.cpp:2194] Processing reply for offers: [ 20140828-212146-16842879-45613-27625-0 ] on slave 20140828-212146-16842879-45613-27625-0 at slave(48)@127.0.1.1:45613 (saucy) for framework 20140828-212146-16842879-45613-27625-0000 I0828 21:21:46.404116 27644 master.cpp:2277] Authorizing framework principal 'test-principal' to launch task cf5afc1b-c007-435b-8c36-be8aa3659d3a as user 'jenkins' I0828 21:21:46.404578 27644 master.hpp:829] Adding task cf5afc1b-c007-435b-8c36-be8aa3659d3a with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140828-212146-16842879-45613-27625-0 (saucy) I0828 21:21:46.404824 27644 master.cpp:2343] Launching task cf5afc1b-c007-435b-8c36-be8aa3659d3a of framework 20140828-212146-16842879-45613-27625-0000 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20140828-212146-16842879-45613-27625-0 at slave(48)@127.0.1.1:45613 (saucy) I0828 21:21:46.405206 27644 slave.cpp:1011] Got assigned task cf5afc1b-c007-435b-8c36-be8aa3659d3a for framework 20140828-212146-16842879-45613-27625-0000 I0828 21:21:46.405462 27644 slave.cpp:3542] Checkpointing FrameworkInfo to '/tmp/SlaveRecoveryTest_0_ShutdownSlave_umhraW/meta/slaves/20140828-212146-16842879-45613-27625-0/frameworks/20140828-212146-16842879-45613-27625-0000/framework.info' I0828 21:21:46.405840 27644 slave.cpp:3549] Checkpointing framework pid 'scheduler-cb5a0264-23cc-45d0-bc4c-a92fa5308158@127.0.1.1:45613' to '/tmp/SlaveRecoveryTest_0_ShutdownSlave_umhraW/meta/slaves/20140828-212146-16842879-45613-27625-0/frameworks/20140828-212146-16842879-45613-27625-0000/framework.pid' I0828 21:21:46.406122 27645 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 7.684731ms I0828 21:21:46.406288 27645 replica.cpp:676] Persisted action at 4 I0828 21:21:46.406618 27645 replica.cpp:655] Replica received learned notice for position 4 I0828 21:21:46.407562 27644 slave.cpp:1121] Launching task cf5afc1b-c007-435b-8c36-be8aa3659d3a for framework 20140828-212146-16842879-45613-27625-0000 I0828 21:21:46.409296 27644 slave.cpp:3858] Checkpointing ExecutorInfo to '/tmp/SlaveRecoveryTest_0_ShutdownSlave_umhraW/meta/slaves/20... | 2 |
| 111 | MESOS-1751 | Request for "stats.json" cannot be fulfilled after stopping the framework | Request for "stats.json" to master from a test case doesn't work after calling frameworks' {{driver.stop()}}. However, it works for "state.json". I think the problem is related to {{stats()}} continuation {{_stats()}}. The following test illustrates the issue: {code:title=TestCase.cpp|borderStyle=solid} TEST_F(MasterTest, RequestAfterDriverStop) { Try<PID<Master> > master = StartMaster(); ASSERT_SOME(master); Try<PID<Slave> > slave = StartSlave(); ASSERT_SOME(slave); MockScheduler sched; MesosSchedulerDriver driver( &sched, DEFAULT_FRAMEWORK_INFO, master.get(), DEFAULT_CREDENTIAL); driver.start(); Future<process::http::Response> response_before = process::http::get(master.get(), "stats.json"); AWAIT_READY(response_before); driver.stop(); Future<process::http::Response> response_after = process::http::get(master.get(), "stats.json"); AWAIT_READY(response_after); driver.join(); Shutdown(); // Must shutdown before 'containerizer' gets deallocated. } {code} | 5 |
| 112 | MESOS-1752 | Allow variadic templates | Add variadic templates to the C++11 configure check. Once there, we can start using them in the code-base. | 1 |
| 113 | MESOS-1758 | Freezer failure leads to lost task during container destruction. | In the past we've seen numerous issues around the freezer. Lately, on the 2.6.44 kernel, we've seen issues where we're unable to freeze the cgroup: (1) An oom occurs. (2) No indication of oom in the kernel logs. (3) The slave is unable to freeze the cgroup. (4) The task is marked as lost. {noformat} I0903 16:46:24.956040 25469 mem.cpp:575] Memory limit exceeded: Requested: 15488MB Maximum Used: 15488MB MEMORY STATISTICS: cache 7958691840 rss 8281653248 mapped_file 9474048 pgpgin 4487861 pgpgout 522933 pgfault 2533780 pgmajfault 11 inactive_anon 0 active_anon 8281653248 inactive_file 7631708160 active_file 326852608 unevictable 0 hierarchical_memory_limit 16240345088 total_cache 7958691840 total_rss 8281653248 total_mapped_file 9474048 total_pgpgin 4487861 total_pgpgout 522933 total_pgfault 2533780 total_pgmajfault 11 total_inactive_anon 0 total_active_anon 8281653248 total_inactive_file 7631728640 total_active_file 326852608 total_unevictable 0 I0903 16:46:24.956848 25469 containerizer.cpp:1041] Container bbb9732a-d600-4c1b-b326-846338c608c3 has reached its limit for resource mem(*):1.62403e+10 and will be terminated I0903 16:46:24.957427 25469 containerizer.cpp:909] Destroying container 'bbb9732a-d600-4c1b-b326-846338c608c3' I0903 16:46:24.958664 25481 cgroups.cpp:2192] Freezing cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 I0903 16:46:34.959529 25488 cgroups.cpp:2209] Thawing cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 I0903 16:46:34.962070 25482 cgroups.cpp:1404] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 after 1.710848ms I0903 16:46:34.962658 25479 cgroups.cpp:2192] Freezing cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 I0903 16:46:44.963349 25488 cgroups.cpp:2209] Thawing cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 I0903 16:46:44.965631 25472 cgroups.cpp:1404] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 after 1.588224ms I0903 16:46:44.966356 25472 cgroups.cpp:2192] Freezing cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 I0903 16:46:54.967254 25488 cgroups.cpp:2209] Thawing cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 I0903 16:46:56.008447 25475 cgroups.cpp:1404] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 after 2.15296ms I0903 16:46:56.009071 25466 cgroups.cpp:2192] Freezing cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 I0903 16:47:06.010329 25488 cgroups.cpp:2209] Thawing cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 I0903 16:47:06.012538 25467 cgroups.cpp:1404] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 after 1.643008ms I0903 16:47:06.013216 25467 cgroups.cpp:2192] Freezing cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 I0903 16:47:12.516348 25480 slave.cpp:3030] Current usage 9.57%. Max allowed age: 5.630238827780799days I0903 16:47:16.015192 25488 cgroups.cpp:2209] Thawing cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 I0903 16:47:16.017043 25486 cgroups.cpp:1404] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 after 1.511168ms I0903 16:47:16.017555 25480 cgroups.cpp:2192] Freezing cgroup /sys/fs/cgroup/freezer/mesos/bbb9732a-d600-4c1b-b326-846338c608c3 I0903 16:47:19.862746 25483 http.cpp:245] HTTP request for '/slave(1)/stats.json' E0903 16:47:24.960055 25472 slave.cpp:2557] Termination of executor 'E' of framework '201104070004-0000002563-0000' failed: Failed to destroy container: discarded future I0903 16:47:24.962054 25472 slave.cpp:2087] Handling status update TASK_LOST (UUID: c0c1633b-7221-40dc-90a2-660ef639f747) for task T of framework 201104070004-0000002563-0000 from @0.0.0.0:0 I0903 16:47:24.963470 25469 mem.cpp:293] Updated 'memory.soft_limit_in_bytes' to 128MB for container bbb9732a-d600-4c1b-b326-846338c608c3 I0903 16:47:24.963541 25471 cpushare.cpp:338] Updated 'cpu.shares' to 256 (cpus 0.25) for container bbb9732a-d600-4c1b-b326-846338c608c3 I0903 16:47:24.964756 25471 cpushare.cpp:359] Updated 'cpu.cfs_period_us' to 100ms and 'cpu.cfs_quota_us' to 25ms (cpus 0.25) for container bbb9732a-d600-4c1b-b326-846338c608c3 I0903 16:47:43.406610 25476 status_update_manager.cpp:320] Received status update TASK_LOST (UUID: c0c1633b-7221-40dc-90a2-660ef639f747) for task T of framework 201104070004-0000002563-0000 I0903 16:47:43.406991 25476 status_update_manager.hpp:342] Checkpointing UPDATE for status update TASK_LOST (UUID: c0c1633b-7221-40dc-90a2-660ef639f747) for task T of framework 201104070004-0000002563-0000 I0903 16:47:43.410475 25476 status_update_manager.cpp:373] Forwarding status update TASK_LOST (UUID: c0c1633b-7221-40dc-90a2-660ef639f747) for task T of framework 201104070004-0000002563-0000 to master@<scrubbed_ip>:5050 I0903 16:47:43.439923 25480 status_update_manager.cpp:398] Received status update acknowledgement (UUID: c0c1633b-7221-40dc-90a2-660ef639f747) for task T of framework 201104070004-0000002563-0000 I0903 16:47:43.440115 25480 status_update_manager.hpp:342] Checkpointing ACK for status update TASK_LOST (UUID: c0c1633b-7221-40dc-90a2-660ef639f747) for task T of framework 201104070004-0000002563-0000 I0903 16:47:43.443595 25480 slave.cpp:2709] Cleaning up executor 'E' of framework 201104070004-0000002563-0000 {noformat} We should consider avoiding the freezer entirely in favor of a kill(2) loop. We don't have to wait for pid namespaces to remove the freezer dependency. At the very least, when the freezer fails, we should proceed with a kill(2) loop to ensure that we destroy the cgroup. | 2 |
| 114 | MESOS-1760 | MasterAuthorizationTest.FrameworkRemovedBeforeReregistration is flaky | Observed this on Apache CI: https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/2355/changes {code} [ RUN] MasterAuthorizationTest.FrameworkRemovedBeforeReregistration Using temporary directory '/tmp/MasterAuthorizationTest_FrameworkRemovedBeforeReregistration_0tw16Z' I0903 22:04:33.520237 25565 leveldb.cpp:176] Opened db in 49.073821ms I0903 22:04:33.538331 25565 leveldb.cpp:183] Compacted db in 18.065051ms I0903 22:04:33.538363 25565 leveldb.cpp:198] Created db iterator in 4826ns I0903 22:04:33.538377 25565 leveldb.cpp:204] Seeked to beginning of db in 682ns I0903 22:04:33.538385 25565 leveldb.cpp:273] Iterated through 0 keys in the db in 312ns I0903 22:04:33.538399 25565 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0903 22:04:33.538624 25593 recover.cpp:425] Starting replica recovery I0903 22:04:33.538707 25598 recover.cpp:451] Replica is in EMPTY status I0903 22:04:33.540909 25590 master.cpp:286] Master 20140903-220433-453759884-44122-25565 (hemera.apache.org) started on 140.211.11.27:44122 I0903 22:04:33.540932 25590 master.cpp:332] Master only allowing authenticated frameworks to register I0903 22:04:33.540936 25590 master.cpp:337] Master only allowing authenticated slaves to register I0903 22:04:33.540941 25590 credentials.hpp:36] Loading credentials for authentication from '/tmp/MasterAuthorizationTest_FrameworkRemovedBeforeReregistration_0tw16Z/credentials' I0903 22:04:33.541337 25590 master.cpp:366] Authorization enabled I0903 22:04:33.541508 25597 replica.cpp:638] Replica in EMPTY status received a broadcasted recover request I0903 22:04:33.542343 25582 hierarchical_allocator_process.hpp:299] Initializing hierarchical allocator process with master : master@140.211.11.27:44122 I0903 22:04:33.542445 25592 master.cpp:120] No whitelist given. Advertising offers for all slaves I0903 22:04:33.543175 25602 recover.cpp:188] Received a recover response from a replica in EMPTY status I0903 22:04:33.543637 25587 recover.cpp:542] Updating replica status to STARTING I0903 22:04:33.544256 25579 master.cpp:1205] The newly elected leader is master@140.211.11.27:44122 with id 20140903-220433-453759884-44122-25565 I0903 22:04:33.544275 25579 master.cpp:1218] Elected as the leading master! I0903 22:04:33.544282 25579 master.cpp:1036] Recovering from registrar I0903 22:04:33.544401 25579 registrar.cpp:313] Recovering registrar I0903 22:04:33.558487 25593 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 14.678563ms I0903 22:04:33.558531 25593 replica.cpp:320] Persisted replica status to STARTING I0903 22:04:33.558653 25593 recover.cpp:451] Replica is in STARTING status I0903 22:04:33.559867 25588 replica.cpp:638] Replica in STARTING status received a broadcasted recover request I0903 22:04:33.560057 25602 recover.cpp:188] Received a recover response from a replica in STARTING status I0903 22:04:33.561280 25584 recover.cpp:542] Updating replica status to VOTING I0903 22:04:33.576900 25581 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 14.712427ms I0903 22:04:33.576942 25581 replica.cpp:320] Persisted replica status to VOTING I0903 22:04:33.577018 25581 recover.cpp:556] Successfully joined the Paxos group I0903 22:04:33.577108 25581 recover.cpp:440] Recover process terminated I0903 22:04:33.577401 25581 log.cpp:656] Attempting to start the writer I0903 22:04:33.578559 25589 replica.cpp:474] Replica received implicit promise request with proposal 1 I0903 22:04:33.594611 25589 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 16.029152ms I0903 22:04:33.594640 25589 replica.cpp:342] Persisted promised to 1 I0903 22:04:33.595391 25584 coordinator.cpp:230] Coordinator attemping to fill missing position I0903 22:04:33.597512 25588 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I0903 22:04:33.613037 25588 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 15.502568ms I0903 22:04:33.613065 25588 replica.cpp:676] Persisted action at 0 I0903 22:04:33.615435 25585 replica.cpp:508] Replica received write request for position 0 I0903 22:04:33.615463 25585 leveldb.cpp:438] Reading position from leveldb took 10743ns I0903 22:04:33.630801 25585 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 15.320225ms I0903 22:04:33.630852 25585 replica.cpp:676] Persisted action at 0 I0903 22:04:33.631126 25585 replica.cpp:655] Replica received learned notice for position 0 I0903 22:04:33.647801 25585 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 16.652951ms I0903 22:04:33.647830 25585 replica.cpp:676] Persisted action at 0 I0903 22:04:33.647842 25585 replica.cpp:661] Replica learned NOP action at position 0 I0903 22:04:33.648548 25583 log.cpp:672] Writer started with ending position 0 I0903 22:04:33.649235 25583 leveldb.cpp:438] Reading position from leveldb took 25209ns I0903 22:04:33.650897 25591 registrar.cpp:346] Successfully fetched the registry (0B) I0903 22:04:33.650930 25591 registrar.cpp:422] Attempting to update the 'registry' I0903 22:04:33.652861 25601 log.cpp:680] Attempting to append 138 bytes to the log I0903 22:04:33.653097 25586 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0903 22:04:33.655225 25590 replica.cpp:508] Replica received write request for position 1 I0903 22:04:33.669618 25590 leveldb.cpp:343] Persisting action (157 bytes) to leveldb took 14.337486ms I0903 22:04:33.669663 25590 replica.cpp:676] Persisted action at 1 I0903 22:04:33.670045 25584 replica.cpp:655] Replica received learned notice for position 1 I0903 22:04:34.414243 25584 leveldb.cpp:343] Persisting action (159 bytes) to leveldb took 15.401247ms I0903 22:04:34.414300 25584 replica.cpp:676] Persisted action at 1 I0903 22:04:34.414316 25584 replica.cpp:661] Replica learned APPEND action at position 1 I0903 22:04:34.414937 25589 registrar.cpp:479] Successfully updated 'registry' I0903 22:04:34.415069 25585 log.cpp:699] Attempting to truncate the log to 1 I0903 22:04:34.415194 25589 registrar.cpp:372] Successfully recovered registrar I0903 22:04:34.415284 25589 master.cpp:1063] Recovered 0 slaves from the Registry (100B) ; allowing 10mins for slaves to re-register I0903 22:04:34.415362 25587 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0903 22:04:34.418926 25597 replica.cpp:508] Replica received write request for position 2 I0903 22:04:34.434321 25597 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 15.368147ms I0903 22:04:34.434352 25597 replica.cpp:676] Persisted action at 2 I0903 22:04:34.435022 25582 replica.cpp:655] Replica received learned notice for position 2 I0903 22:04:34.450331 25582 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 15.284486ms I0903 22:04:34.450387 25582 leveldb.cpp:401] Deleting ~1 keys from leveldb took 25774ns I0903 22:04:34.450402 25582 replica.cpp:676] Persisted action at 2 I0903 22:04:34.450412 25582 replica.cpp:661] Replica learned TRUNCATE action at position 2 I0903 22:04:34.460691 25565 sched.cpp:137] Version: 0.21.0 I0903 22:04:34.460927 25582 sched.cpp:233] New master detected at master@140.211.11.27:44122 I0903 22:04:34.460948 25582 sched.cpp:283] Authenticating with master master@140.211.11.27:44122 I0903 22:04:34.461359 25582 authenticatee.hpp:128] Creating new client SASL connection I0903 22:04:34.461647 25582 master.cpp:3637] Authenticating scheduler-04e0b571-7e0c-4ef3-bb14-c6bbfd8ac9a4@140.211.11.27:44122 I0903 22:04:34.461801 25598 authenticator.hpp:156] Creating new server SASL connection I0903 22:04:34.462172 25598 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0903 22:04:34.462185 25598 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0903 22:04:34.462257 25598 authenticator.hpp:262] Received SASL authentication start I0903 22:04:34.462323 25598 authenticator.hpp:384] Authentication requires more steps I0903 22:04:34.462345 25598 authenticatee.hpp:265] Received SASL authentication step I0903 22:04:34.462417 25598 authenticator.hpp:290] Received SASL authentication step I0903 22:04:34.462522 25598 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'hemera.apache.org' server FQDN: 'hemera.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0903 22:04:34.462529 25598 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0903 22:04:34.462538 25598 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0903 22:04:34.462543 25598 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'hemera.apache.org' server FQDN: 'hemera.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0903 22:04:34.462548 25598 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0903 22:04:34.462550 25598 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0903 22:04:34.462558 25598 authenticator.hpp:376] Authentication success I0903 22:04:34.462635 25598 master.cpp:3677] Successfully authenticated principal 'test-principal' at scheduler-04e0b571-7e0c-4ef3-bb14-c6bbfd8ac9a4@140.211.11.27:44122 I0903 22:04:34.462687 25590 authenticatee.hpp:305] Authentication success I0903 22:04:34.463219 25588 sched.cpp:357] Successfully authenticated with master master@140.211.11.27:44122 I0903 22:04:34.463243 25588 sched.cpp:476] Sending registration request to master@140.211.11.27:44122 I0903 22:04:34.463307 25588 master.cpp:1324] Received registration request from scheduler-04e0b571-7e0c-4ef3-bb14-c6bbfd8ac9a4@140.211.11.27:44122 I0903 22:04:34.463330 25588 master.cpp:1284] Authorizing framework principal 'test-principal' to receive offers for role '*' I0903 22:04:34.463412 25588 master.cpp:1383] Registering framework 20140903-220433-453759884-44122-25565-0000 at scheduler-04e0b571-7e0c-4ef3-bb14-c6bbfd8ac9a4@140.211.11.27:44122 I0903 22:04:34.463577 25598 sched.cpp:407] Framework registered with 20140903-220433-453759884-44122-25565-0000 I0903 22:04:34.463728 25587 hierarchical_allocator_process.hpp:329] Added framework 20140903-220433-453759884-44122-25565-0000 I0903 22:04:34.463739 25587 hierarchical_allocator_process.hpp:697] No resources available to allocate! I0903 22:04:34.463743 25587 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 5016ns I0903 22:04:34.463755 25598 sched.cpp:421] Scheduler::registered took 165035ns I0903 22:04:34.465558 25583 sched.cpp:227] Scheduler::disconnected took 6254ns I0903 22:04:34.465566 25583 sched.cpp:233] New master detected at master@140.211.11.27:44122 I0903 22:04:34.465575 25583 sched.cpp:283] Authenticating with master master@140.211.11.27:44122 I0903 22:04:34.465642 25583 authenticatee.hpp:128] Creating new client SASL connection I0903 22:04:34.465790 25583 master.cpp:1680] Deactivating framework 20140903-220433-453759884-44122-25565-0000 I0903 22:04:34.465850 25583 master.cpp:3637] Authenticating scheduler-04e0b571-7e0c-4ef3-bb14-c6bbfd8ac9a4@140.211.11.27:44122 I0903 22:04:34.465879 25601 hierarchical_allocator_process.hpp:405] Deactivated framework 20140903-220433-453759884-44122-25565-0000 I0903 22:04:34.466047 25600 authenticator.hpp:156] Creating new server SASL connection I0903 22:04:34.466315 25600 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0903 22:04:34.466326 25600 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0903 22:04:34.466346 25600 authenticator.hpp:262] Received SASL authentication start I0903 22:04:34.466418 25600 authenticator.hpp:384] Authentication requires more steps I0903 22:04:34.466436 25600 authenticatee.hpp:265] Received SASL authentication step I0903 22:04:34.466475 25600 authenticator.hpp:290] Received SASL authentication step I0903 22:04:34.466486 25600 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'hemera.apache.org' server FQDN: 'hemera.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0903 22:04:34.466491 25600 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0903 22:04:34.466496 25600 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0903 22:04:34.466502 25600 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'hemera.apache.org' server FQDN: 'hemera.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0903 22:04:34.466506 25600 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0903 22:04:34.466509 25600 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0903 22:04:34.466516 25600 authenticator.hpp:376] Authentication success I0903 22:04:34.466596 25588 master.cpp:3677] Successfully authenticated principal 'test-principal' at scheduler-04e0b571-7e0c-4ef3-bb14-c6bbfd8ac9a4@140.211.11.27:44122 I0903 22:04:34.466629 25597 authenticatee.hpp:305] Authentication success I0903 22:04:34.467062 25594 sched.cpp:357] Successfully authenticated with master master@140.211.11.27:44122 I0903 22:04:34.467077 25594 sched.cpp:476] Sending registration request to master@140.211.11.27:44122 I0903 22:04:34.467190 25588 master.cpp:1448] Received re-registration request from framework 20140903-220433-453759884-44122-25565-0000 at scheduler-04e0b571-7e0c-4ef3-bb14-c6bbfd8ac9a4@140.211.11.27:44122 I0903 22:04:36.368134 25588 master.cpp:1284] Authorizing framework principal 'test-principal' to receive offers for role '*' I0903 22:04:34.542999 25594 hierarchical_allocator_process.hpp:697] No resources available to allocate! I0903 22:04:35.463639 25582 sched.cpp:476] Sending registration request to master@140.211.11.27:44122 I0903 22:04:36.368185 25594 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 1.825177748secs I0903 22:04:36.368302 25588 master.cpp:1448] Received re-registration request from framework 20140903-220433-453759884-44122-25565-0000 at scheduler-04e0b571-7e0c-4ef3-bb14-c6bbfd8ac9a4@140.211.11.27:44122 I0903 22:04:36.368330 25588 master.cpp:1284] Authorizing framework principal 'test-principal' to receive offers for role '*' I0903 22:04:36.368388 25582 sched.cpp:476] Sending registration request to master@140.211.11.27:44122 : Failure Mock function called more times than expected - returning default value. Function call: authorize(@0x2ba11964c1b0 40-byte object <D0-ED 39-16 A1-2B 00-00 00-00 00-00 00-00 00-00 00-6C 01-3C A1-2B 00-00 30-20 00-3C A1-2B 00-00 00-00 00-00 03-00 00-00>) The mock function has no default action set, and its return type has no default value set. *** Aborted at 1409781876 (unix time) try "date -d @1409781876" if you are using GNU date *** I0903 22:04:36.368913 25598 sched.cpp:745] Stopping framework '20140903-220433-453759884-44122-25565-0000' PC: @ 0x2ba117a990d5 (unknown) *** SIGABRT (@0x3ea000063dd) received by PID 25565 (TID 0x2ba11964d700) from PID 25565; stack trace: *** @ 0x2ba117854cb0 (unknown) @ 0x2ba117a990d5 (unknown) @ 0x2ba117a9c83b (unknown) @ 0x9cba9d testing::internal::GoogleTestFailureReporter::ReportFailure() @ 0x790091 testing::internal::FunctionMockerBase<>::PerformDefaultAction() @ 0x790166 testing::internal::FunctionMockerBase<>::UntypedPerformDefaultAction() @ 0x9c3daa testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith() @ 0x787279 mesos::internal::tests::MockAuthorizer::authorize() @ 0x2ba1157c133d mesos::internal::master::Master::validate() @ 0x2ba1157c2b7a mesos::internal::master::Master::reregisterFramework() @ 0x2ba1157e0038 ProtobufProcess<>::handler2<>() @ 0x2ba1157dde89 std::tr1::_Function_handler<>::_M_invoke() @ 0x2ba1157b15f7 mesos::internal::master::Master::_visit() @ 0x2ba1157bfa3e mesos::internal::master::Master::visit() @ 0x2ba115caf5e7 process::ProcessManager::resume() @ 0x2ba115cb027c process::schedule() @ 0x2ba11784ce9a start_thread @ 0x2ba117b5731d (unknown) {code} | 1 |
| 115 | MESOS-1765 | Use PID namespace to avoid freezing cgroup | There is some known kernel issue when we freeze the whole cgroup upon OOM. Mesos probably can just use PID namespace so that we will only need to kill the "init" of the pid namespace, instead of freezing all the processes and killing them one by one. But I am not quite sure if this would break the existing code. | 5 |
| 116 | MESOS-1766 | MasterAuthorizationTest.DuplicateRegistration test is flaky | {code} [ RUN ] MasterAuthorizationTest.DuplicateRegistration Using temporary directory '/tmp/MasterAuthorizationTest_DuplicateRegistration_pVJg7m' I0905 15:53:16.398993 25769 leveldb.cpp:176] Opened db in 2.601036ms I0905 15:53:16.399566 25769 leveldb.cpp:183] Compacted db in 546216ns I0905 15:53:16.399590 25769 leveldb.cpp:198] Created db iterator in 2787ns I0905 15:53:16.399605 25769 leveldb.cpp:204] Seeked to beginning of db in 500ns I0905 15:53:16.399617 25769 leveldb.cpp:273] Iterated through 0 keys in the db in 185ns I0905 15:53:16.399633 25769 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0905 15:53:16.399817 25786 recover.cpp:425] Starting replica recovery I0905 15:53:16.399952 25793 recover.cpp:451] Replica is in EMPTY status I0905 15:53:16.400683 25795 replica.cpp:638] Replica in EMPTY status received a broadcasted recover request I0905 15:53:16.400795 25787 recover.cpp:188] Received a recover response from a replica in EMPTY status I0905 15:53:16.401005 25783 recover.cpp:542] Updating replica status to STARTING I0905 15:53:16.401470 25786 master.cpp:286] Master 20140905-155316-3125920579-49188-25769 (penates.apache.org) started on 67.195.81.186:49188 I0905 15:53:16.401521 25786 master.cpp:332] Master only allowing authenticated frameworks to register I0905 15:53:16.401533 25786 master.cpp:337] Master only allowing authenticated slaves to register I0905 15:53:16.401543 25786 credentials.hpp:36] Loading credentials for authentication from '/tmp/MasterAuthorizationTest_DuplicateRegistration_pVJg7m/credentials' I0905 15:53:16.401558 25793 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 474683ns I0905 15:53:16.401582 25793 replica.cpp:320] Persisted replica status to STARTING I0905 15:53:16.401667 25793 recover.cpp:451] Replica is in STARTING status I0905 15:53:16.401669 25786 master.cpp:366] Authorization enabled I0905 15:53:16.401898 25795 master.cpp:120] No whitelist given. Advertising offers for all slaves I0905 15:53:16.401936 25796 hierarchical_allocator_process.hpp:299] Initializing hierarchical allocator process with master : master@67.195.81.186:49188 I0905 15:53:16.402160 25784 replica.cpp:638] Replica in STARTING status received a broadcasted recover request I0905 15:53:16.402333 25790 master.cpp:1205] The newly elected leader is master@67.195.81.186:49188 with id 20140905-155316-3125920579-49188-25769 I0905 15:53:16.402359 25790 master.cpp:1218] Elected as the leading master! I0905 15:53:16.402371 25790 master.cpp:1036] Recovering from registrar I0905 15:53:16.402472 25798 registrar.cpp:313] Recovering registrar I0905 15:53:16.402529 25791 recover.cpp:188] Received a recover response from a replica in STARTING status I0905 15:53:16.402782 25788 recover.cpp:542] Updating replica status to VOTING I0905 15:53:16.403002 25795 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 116403ns I0905 15:53:16.403020 25795 replica.cpp:320] Persisted replica status to VOTING I0905 15:53:16.403081 25791 recover.cpp:556] Successfully joined the Paxos group I0905 15:53:16.403197 25791 recover.cpp:440] Recover process terminated I0905 15:53:16.403388 25796 log.cpp:656] Attempting to start the writer I0905 15:53:16.403993 25784 replica.cpp:474] Replica received implicit promise request with proposal 1 I0905 15:53:16.404147 25784 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 132156ns I0905 15:53:16.404167 25784 replica.cpp:342] Persisted promised to 1 I0905 15:53:16.404542 25795 coordinator.cpp:230] Coordinator attemping to fill missing position I0905 15:53:16.405498 25787 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I0905 15:53:16.405868 25787 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 347231ns I0905 15:53:16.405886 25787 replica.cpp:676] Persisted action at 0 I0905 15:53:16.406553 25788 replica.cpp:508] Replica received write request for position 0 I0905 15:53:16.406582 25788 leveldb.cpp:438] Reading position from leveldb took 11402ns I0905 15:53:16.529067 25788 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 535803ns I0905 15:53:16.529088 25788 replica.cpp:676] Persisted action at 0 I0905 15:53:16.529355 25784 replica.cpp:655] Replica received learned notice for position 0 I0905 15:53:16.529784 25784 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 406036ns I0905 15:53:16.529806 25784 replica.cpp:676] Persisted action at 0 I0905 15:53:16.529817 25784 replica.cpp:661] Replica learned NOP action at position 0 I0905 15:53:16.530108 25783 log.cpp:672] Writer started with ending position 0 I0905 15:53:16.530597 25792 leveldb.cpp:438] Reading position from leveldb took 14594ns I0905 15:53:16.532060 25787 registrar.cpp:346] Successfully fetched the registry (0B) I0905 15:53:16.532091 25787 registrar.cpp:422] Attempting to update the 'registry' I0905 15:53:16.533537 25785 log.cpp:680] Attempting to append 140 bytes to the log I0905 15:53:16.533596 25785 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0905 15:53:16.533998 25798 replica.cpp:508] Replica received write request for position 1 I0905 15:53:16.534397 25798 leveldb.cpp:343] Persisting action (159 bytes) to leveldb took 372452ns I0905 15:53:16.534416 25798 replica.cpp:676] Persisted action at 1 I0905 15:53:16.534808 25793 replica.cpp:655] Replica received learned notice for position 1 I0905 15:53:16.534996 25793 leveldb.cpp:343] Persisting action (161 bytes) to leveldb took 164609ns I0905 15:53:16.535014 25793 replica.cpp:676] Persisted action at 1 I0905 15:53:16.535025 25793 replica.cpp:661] Replica learned APPEND action at position 1 I0905 15:53:16.535368 25784 registrar.cpp:479] Successfully updated 'registry' I0905 15:53:16.535419 25784 registrar.cpp:372] Successfully recovered registrar I0905 15:53:16.535452 25785 log.cpp:699] Attempting to truncate the log to 1 I0905 15:53:16.535555 25791 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0905 15:53:16.535553 25792 master.cpp:1063] Recovered 0 slaves from the Registry (102B) ; allowing 10mins for slaves to re-register I0905 15:53:16.536038 25784 replica.cpp:508] Replica received write request for position 2 I0905 15:53:16.536166 25784 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 101619ns I0905 15:53:16.536185 25784 replica.cpp:676] Persisted action at 2 I0905 15:53:16.536497 25791 replica.cpp:655] Replica received learned notice for position 2 I0905 15:53:16.536633 25791 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 109281ns I0905 15:53:16.536664 25791 leveldb.cpp:401] Deleting ~1 keys from leveldb took 14164ns I0905 15:53:16.536677 25791 replica.cpp:676] Persisted action at 2 I0905 15:53:16.536689 25791 replica.cpp:661] Replica learned TRUNCATE action at position 2 I0905 15:53:16.548408 25769 sched.cpp:137] Version: 0.21.0 I0905 15:53:16.548627 25792 sched.cpp:233] New master detected at master@67.195.81.186:49188 I0905 15:53:16.548653 25792 sched.cpp:283] Authenticating with master master@67.195.81.186:49188 I0905 15:53:16.548857 25797 authenticatee.hpp:128] Creating new client SASL connection I0905 15:53:16.548950 25797 master.cpp:3637] Authenticating scheduler-33430370-6af5-4c7b-bbd8-f6a43269ecf5@67.195.81.186:49188 I0905 15:53:16.549041 25797 authenticator.hpp:156] Creating new server SASL connection I0905 15:53:16.549120 25797 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0905 15:53:16.549141 25797 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0905 15:53:16.549180 25797 authenticator.hpp:262] Received SASL authentication start I0905 15:53:16.549229 25797 authenticator.hpp:384] Authentication requires more steps I0905 15:53:16.549268 25797 authenticatee.hpp:265] Received SASL authentication step I0905 15:53:16.549351 25787 authenticator.hpp:290] Received SASL authentication step I0905 15:53:16.549378 25787 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'penates.apache.org' server FQDN: 'penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0905 15:53:16.549391 25787 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0905 15:53:16.549403 25787 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0905 15:53:16.549415 25787 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'penates.apache.org' server FQDN: 'penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0905 15:53:16.549424 25787 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0905 15:53:16.549432 25787 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0905 15:53:16.549448 25787 authenticator.hpp:376] Authentication success I0905 15:53:16.549489 25787 authenticatee.hpp:305] Authentication success I0905 15:53:16.549525 25787 master.cpp:3677] Successfully authenticated principal 'test-principal' at scheduler-33430370-6af5-4c7b-bbd8-f6a43269ecf5@67.195.81.186:49188 I0905 15:53:16.549669 25783 sched.cpp:357] Successfully authenticated with master master@67.195.81.186:49188 I0905 15:53:16.549690 25783 sched.cpp:476] Sending registration request to master@67.195.81.186:49188 I0905 15:53:16.549751 25787 master.cpp:1324] Received registration request from scheduler-33430370-6af5-4c7b-bbd8-f6a43269ecf5@67.195.81.186:49188 I0905 15:53:16.549782 25787 master.cpp:1284] Authorizing framework principal 'test-principal' to receive offers for role '*' I0905 15:53:16.551250 25791 sched.cpp:233] New master detected at master@67.195.81.186:49188 I0905 15:53:16.551273 25791 sched.cpp:283] Authenticating with master master@67.195.81.186:49188 I0905 15:53:16.551357 25788 authenticatee.hpp:128] Creating new client SASL connection I0905 15:53:16.551456 25791 master.cpp:3637] Authenticating scheduler-33430370-6af5-4c7b-bbd8-f6a43269ecf5@67.195.81.186:49188 I0905 15:53:16.551553 25788 authenticator.hpp:156] Creating new server SASL connection I0905 15:53:16.551673 25786 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0905 15:53:16.551697 25786 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0905 15:53:16.551755 25792 authenticator.hpp:262] Received SASL authentication start I0905 15:53:16.551808 25792 authenticator.hpp:384] Authentication requires more steps I0905 15:53:16.551856 25792 authenticatee.hpp:265] Received SASL authentication step I0905 15:53:16.551920 25786 authenticator.hpp:290] Received SASL authentication step I0905 15:53:16.551949 25786 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'penates.apache.org' server FQDN: 'penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0905 15:53:16.551966 25786 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0905 15:53:16.551985 25786 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0905 15:53:16.551997 25786 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'penates.apache.org' server FQDN: 'penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0905 15:53:16.552006 25786 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0905 15:53:16.552014 25786 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0905 15:53:16.552031 25786 authenticator.hpp:376] Authentication success I0905 15:53:16.552081 25792 authenticatee.hpp:305] Authentication success I0905 15:53:16.552100 25786 master.cpp:3677] Successfully authenticated principal 'test-principal' at scheduler-33430370-6af5-4c7b-bbd8-f6a43269ecf5@67.195.81.186:49188 I0905 15:53:16.552249 25792 sched.cpp:357] Successfully authenticated with master master@67.195.81.186:49188 I0905 15:53:17.402861 25793 hierarchical_allocator_process.hpp:697] No resources available to allocate! I0905 15:53:18.874348 25792 sched.cpp:476] Sending registration request to master@67.195.81.186:49188 I0905 15:53:18.874364 25793 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 1.471501003secs I0905 15:53:18.874420 25792 sched.cpp:476] Sending registration request to master@67.195.81.186:49188 I0905 15:53:18.874451 25793 master.cpp:1324] Received registration request from scheduler-33430370-6af5-4c7b-bbd8-f6a43269ecf5@67.195.81.186:49188 I0905 15:53:18.874480 25793 master.cpp:1284] Authorizing framework principal 'test-principal' to receive offers for role '*' I0905 15:53:18.874565 25793 master.cpp:1324] Received registration request from scheduler-33430370-6af5-4c7b-bbd8-f6a43269ecf5@67.195.81.186:49188 I0905 15:53:18.874588 25793 master.cpp:1284] Authorizing framework principal 'test-principal' to receive offers for role '*' : Failure Mock function called more times than expected - returning default value. Function call: authorize(@0x2b9ed7fe9350 40-byte object <90-BA B4-D4 9E-2B 00-00 00-00 00-00 00-00 00-00 A0-FA 06-F4 9E-2B 00-00 80-17 09-F4 9E-2B 00-00 00-00 00-00 03-00 00-00>) The mock function has no default action set, and its return type has no default value set. *** Aborted at 1409932398 (unix time) try "date -d @1409932398" if you are using GNU date *** PC: @ 0x2b9ed6233f79 (unknown) *** SIGABRT (@0x95c000064a9) received by PID 25769 (TID 0x2b9ed7fea700) from PID 25769; stack trace: *** @ 0x2b9ed5fef340 (unknown) @ 0x2b9ed6233f79 (unknown) @ 0x2b9ed6237388 (unknown) @ 0x93a5ec testing::internal::GoogleTestFailureReporter::ReportFailure() @ 0x7296c5 testing::internal::FunctionMockerBase<>::UntypedPerformDefaultAction() @ 0x933094 testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith() @ 0x71fbde mesos::internal::tests::MockAuthorizer::authorize() @ 0x2b9ed4038caf mesos::internal::master::Master::validate() @ 0x2b9ed4039763 mesos::internal::master::Master::registerFramework() @ 0x2b9ed40a0c0f ProtobufProcess<>::handler1<>() @ 0x2b9ed4050c57 std::_Function_handler<>::_M_invoke() @ 0x2b9ed407d202 ProtobufProcess<>::visit() @ 0x2b9ed402af1a mesos::internal::master::Master::_visit() @ 0x2b9ed4037eb8 mesos::internal::master::Master::visit() @ 0x2b9ed44cb792 process::ProcessManager::resume() @ 0x2b9ed44cba9c process::schedule() @ 0x2b9ed5fe7182 start_thread @ 0x2b9ed62f830d (unknown) {code} | 2 |
| 117 | MESOS-1771 | introduce unique_ptr | * add unique_ptr to the configure check * document use of unique_ptr in style guide ** use when possible, use std::move when necessary * move raw pointers to Owned to establish ownership * deprecate Owned in favour of unique_ptr | 1 |
| 118 | MESOS-1777 | Design persistent resources | NULL | 13 |
| 119 | MESOS-1778 | Provide an option to validate flag value in stout/flags. | Currently we can provide the default value for a flag, but cannot check if the flag is set to a reasonable value and, e.g., issue a warning. Passing an optional lambda checker to {{FlagBase::add()}} can be a possible solution. | 3 |
| 120 | MESOS-1782 | AllocatorTest/0.FrameworkExited is flaky | {noformat:title=} [ RUN ] AllocatorTest/0.FrameworkExited Using temporary directory '/tmp/AllocatorTest_0_FrameworkExited_B6WZng' I0909 08:02:35.116555 18112 leveldb.cpp:176] Opened db in 31.64686ms I0909 08:02:35.126065 18112 leveldb.cpp:183] Compacted db in 9.449823ms I0909 08:02:35.126118 18112 leveldb.cpp:198] Created db iterator in 5858ns I0909 08:02:35.126137 18112 leveldb.cpp:204] Seeked to beginning of db in 1136ns I0909 08:02:35.126150 18112 leveldb.cpp:273] Iterated through 0 keys in the db in 560ns I0909 08:02:35.126178 18112 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0909 08:02:35.126502 18133 recover.cpp:425] Starting replica recovery I0909 08:02:35.126601 18133 recover.cpp:451] Replica is in EMPTY status I0909 08:02:35.127012 18133 replica.cpp:638] Replica in EMPTY status received a broadcasted recover request I0909 08:02:35.127094 18133 recover.cpp:188] Received a recover response from a replica in EMPTY status I0909 08:02:35.127223 18133 recover.cpp:542] Updating replica status to STARTING I0909 08:02:35.226631 18133 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 99.308134ms I0909 08:02:35.226690 18133 replica.cpp:320] Persisted replica status to STARTING I0909 08:02:35.226812 18131 recover.cpp:451] Replica is in STARTING status I0909 08:02:35.227246 18131 replica.cpp:638] Replica in STARTING status received a broadcasted recover request I0909 08:02:35.227308 18131 recover.cpp:188] Received a recover response from a replica in STARTING status I0909 08:02:35.227409 18131 recover.cpp:542] Updating replica status to VOTING I0909 08:02:35.228540 18129 master.cpp:286] Master 20140909-080235-16842879-44005-18112 (precise) started on 127.0.1.1:44005 I0909 08:02:35.228593 18129 master.cpp:332] Master only allowing authenticated frameworks to register I0909 08:02:35.228607 18129 master.cpp:337] Master only allowing authenticated slaves to register I0909 08:02:35.228620 18129 credentials.hpp:36] Loading credentials for authentication from '/tmp/AllocatorTest_0_FrameworkExited_B6WZng/credentials' I0909 08:02:35.228754 18129 master.cpp:366] Authorization enabled I0909 08:02:35.229560 18129 master.cpp:120] No whitelist given. Advertising offers for all slaves I0909 08:02:35.229933 18129 hierarchical_allocator_process.hpp:299] Initializing hierarchical allocator process with master : master@127.0.1.1:44005 I0909 08:02:35.230057 18127 master.cpp:1212] The newly elected leader is master@127.0.1.1:44005 with id 20140909-080235-16842879-44005-18112 I0909 08:02:35.230129 18127 master.cpp:1225] Elected as the leading master! I0909 08:02:35.230144 18127 master.cpp:1043] Recovering from registrar I0909 08:02:35.230257 18127 registrar.cpp:313] Recovering registrar I0909 08:02:35.232461 18131 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 4.999384ms I0909 08:02:35.232489 18131 replica.cpp:320] Persisted replica status to VOTING I0909 08:02:35.232544 18131 recover.cpp:556] Successfully joined the Paxos group I0909 08:02:35.232611 18131 recover.cpp:440] Recover process terminated I0909 08:02:35.232727 18131 log.cpp:656] Attempting to start the writer I0909 08:02:35.233012 18131 replica.cpp:474] Replica received implicit promise request with proposal 1 I0909 08:02:35.238785 18131 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 5.749504ms I0909 08:02:35.238818 18131 replica.cpp:342] Persisted promised to 1 I0909 08:02:35.244056 18131 coordinator.cpp:230] Coordinator attemping to fill missing position I0909 08:02:35.244580 18131 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I0909 08:02:35.250143 18131 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 5.382351ms I0909 08:02:35.250319 18131 replica.cpp:676] Persisted action at 0 I0909 08:02:35.250901 18131 replica.cpp:508] Replica received write request for position 0 I0909 08:02:35.251137 18131 leveldb.cpp:438] Reading position from leveldb took 18689ns I0909 08:02:35.256597 18131 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 5.274169ms I0909 08:02:35.256764 18131 replica.cpp:676] Persisted action at 0 I0909 08:02:35.263712 18126 replica.cpp:655] Replica received learned notice for position 0 I0909 08:02:35.269613 18126 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 5.417225ms I0909 08:02:35.351641 18126 replica.cpp:676] Persisted action at 0 I0909 08:02:35.351655 18126 replica.cpp:661] Replica learned NOP action at position 0 I0909 08:02:35.351889 18126 log.cpp:672] Writer started with ending position 0 I0909 08:02:35.352165 18126 leveldb.cpp:438] Reading position from leveldb took 25215ns I0909 08:02:35.353163 18126 registrar.cpp:346] Successfully fetched the registry (0B) I0909 08:02:35.353185 18126 registrar.cpp:422] Attempting to update the 'registry' I0909 08:02:35.354152 18126 log.cpp:680] Attempting to append 120 bytes to the log I0909 08:02:35.354195 18126 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0909 08:02:35.354416 18126 replica.cpp:508] Replica received write request for position 1 I0909 08:02:35.351579 18127 hierarchical_allocator_process.hpp:697] No resources available to allocate! I0909 08:02:35.354558 18127 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 2.984795ms I0909 08:02:35.360254 18126 leveldb.cpp:343] Persisting action (137 bytes) to leveldb took 5.811986ms I0909 08:02:35.360285 18126 replica.cpp:676] Persisted action at 1 I0909 08:02:35.364126 18132 replica.cpp:655] Replica received learned notice for position 1 I0909 08:02:35.369856 18132 leveldb.cpp:343] Persisting action (139 bytes) to leveldb took 5.702756ms I0909 08:02:35.369899 18132 replica.cpp:676] Persisted action at 1 I0909 08:02:35.369910 18132 replica.cpp:661] Replica learned APPEND action at position 1 I0909 08:02:35.370209 18132 registrar.cpp:479] Successfully updated 'registry' I0909 08:02:35.370311 18132 registrar.cpp:372] Successfully recovered registrar I0909 08:02:35.370477 18132 log.cpp:699] Attempting to truncate the log to 1 I0909 08:02:35.370553 18132 master.cpp:1070] Recovered 0 slaves from the Registry (84B) ; allowing 10mins for slaves to re-register I0909 08:02:35.370594 18132 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0909 08:02:35.371201 18127 replica.cpp:508] Replica received write request for position 2 I0909 08:02:35.376760 18127 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 5.264501ms I0909 08:02:35.377105 18127 replica.cpp:676] Persisted action at 2 I0909 08:02:35.377770 18127 replica.cpp:655] Replica received learned notice for position 2 I0909 08:02:35.383363 18127 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 5.272769ms I0909 08:02:35.383818 18127 leveldb.cpp:401] Deleting ~1 keys from leveldb took 28148ns I0909 08:02:35.384137 18127 replica.cpp:676] Persisted action at 2 I0909 08:02:35.384399 18127 replica.cpp:661] Replica learned TRUNCATE action at position 2 I0909 08:02:35.396512 18127 slave.cpp:167] Slave started on 64)@127.0.1.1:44005 I0909 08:02:35.654770 18131 hierarchical_allocator_process.hpp:697] No resources available to allocate! I0909 08:02:35.654847 18131 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 104933ns I0909 08:02:35.654974 18127 credentials.hpp:84] Loading credential for authentication from '/tmp/AllocatorTest_0_FrameworkExited_xV9Mk4/credential' I0909 08:02:35.655097 18127 slave.cpp:274] Slave using credential for: test-principal I0909 08:02:35.655203 18127 slave.cpp:287] Slave resources: cpus(*):3; mem(*):1024; disk(*):25116; ports(*):[31000-32000] I0909 08:02:35.655274 18127 slave.cpp:315] Slave hostname: precise I0909 08:02:35.655285 18127 slave.cpp:316] Slave checkpoint: false I0909 08:02:35.655804 18127 state.cpp:33] Recovering state from '/tmp/AllocatorTest_0_FrameworkExited_xV9Mk4/meta' I0909 08:02:35.655913 18127 status_update_manager.cpp:193] Recovering status update manager I0909 08:02:35.656005 18127 slave.cpp:3202] Finished recovery I0909 08:02:35.656251 18127 slave.cpp:598] New master detected at master@127.0.1.1:44005 I0909 08:02:35.656285 18127 slave.cpp:672] Authenticating with master master@127.0.1.1:44005 I0909 08:02:35.656325 18127 slave.cpp:645] Detecting new master I0909 08:02:35.656358 18127 status_update_manager.cpp:167] New master detected at master@127.0.1.1:44005 I0909 08:02:35.656389 18127 authenticatee.hpp:128] Creating new client SASL connection I0909 08:02:35.656563 18127 master.cpp:3653] Authenticating slave(64)@127.0.1.1:44005 I0909 08:02:35.656651 18127 authenticator.hpp:156] Creating new server SASL connection I0909 08:02:35.656770 18127 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0909 08:02:35.656796 18127 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0909 08:02:35.656822 18127 authenticator.hpp:262] Received SASL authentication start I0909 08:02:35.656858 18127 authenticator.hpp:384] Authentication requires more steps I0909 08:02:35.656883 18127 authenticatee.hpp:265] Received SASL authentication step I0909 08:02:35.656924 18127 authenticator.hpp:290] Received SASL authentication step I0909 08:02:35.656960 18127 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'precise' server FQDN: 'precise' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0909 08:02:35.656971 18127 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0909 08:02:35.656982 18127 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0909 08:02:35.656997 18127 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'precise' server FQDN: 'precise' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0909 08:02:35.657004 18127 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0909 08:02:35.657008 18127 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0909 08:02:35.657019 18127 authenticator.hpp:376] Authentication success I0909 08:02:35.657047 18127 authenticatee.hpp:305] Authentication success I0909 08:02:35.657073 18127 master.cpp:3693] Successfully authenticated principal 'test-principal' at slave(64)@127.0.1.1:44005 I0909 08:02:35.657145 18127 slave.cpp:729] Successfully authenticated with master master@127.0.1.1:44005 I0909 08:02:35.657183 18127 slave.cpp:980] Will retry registration in 19.238717ms if necessary I0909 08:02:35.657276 18128 master.cpp:2843] Registering slave at slave(64)@127.0.1.1:44005 (precise) with id 20140909-080235-16842879-44005-18112-0 I0909 08:02:35.657389 18128 registrar.cpp:422] Attempting to update the 'registry' I0909 08:02:35.658382 18130 log.cpp:680] Attempting to append 295 bytes to the log I0909 08:02:35.658432 18130 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I0909 08:02:35.658635 18130 replica.cpp:508] Replica received write request for position 3 I0909 08:02:35.660959 18112 sched.cpp:137] Version: 0.21.0 I0909 08:02:35.661093 18126 sched.cpp:233] New master detected at master@127.0.1.1:44005 I0909 08:02:35.661111 18126 sched.cpp:283] Authenticating with master master@127.0.1.1:44005 I0909 08:02:35.661175 18126 authenticatee.hpp:128] Creating new client SASL connection I0909 08:02:35.661306 18126 master.cpp:3653] Authenticating scheduler-fd929918-7057-4fef-923a-ed9d6fd355be@127.0.1.1:44005 I0909 08:02:35.661376 18126 authenticator.hpp:156] Creating new server SASL connection I0909 08:02:35.661466 18126 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0909 08:02:35.661483 18126 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0909 08:02:35.661504 18126 authenticator.hpp:262] Received SASL authentication start I0909 08:02:35.661530 18126 authenticator.hpp:384] Authentication requires more steps I0909 08:02:35.661552 18126 authenticatee.hpp:265] Received SASL authentication step I0909 08:02:35.661579 18126 authenticator.hpp:290] Received SASL authentication step I0909 08:02:35.661592 18126 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'precise' server FQDN: 'precise' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0909 08:02:35.661598 18126 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0909 08:02:35.661607 18126 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0909 08:02:35.661613 18126 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'precise' server FQDN: 'precise' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0909 08:02:35.661619 18126 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0909 08:02:35.661623 18126 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0909 08:02:35.661633 18126 authenticator.hpp:376] Authentication success I0909 08:02:35.661653 18126 authenticatee.hpp:305] Authentication success I0909 08:02:35.661672 18126 master.cpp:3693] Successfully authenticated principal 'test-principal' at scheduler-fd929918-7057-4fef-923a-ed9d6fd355be@127.0.1.1:44005 I0909 08:02:35.661730 18126 sched.cpp:357] Successfully authenticated with master master@127.0.1.1:44005 I0909 08:02:35.661741 18126 sched.cpp:476] Sending registration request to master@127.0.1.1:44005 I0909 08:02:35.661782 18126 master.cpp:1331] Received registration request from scheduler-fd929918-7057-4fef-923a-ed9d6fd355be@127.0.1.1:44005 I0909 08:02:35.661798 18126 master.cpp:1291] Authorizing framework principal 'test-principal' to receive offers for role '*' I0909 08:02:35.661917 18126 master.cpp:1390] Registering framework 20140909-080235-16842879-44005-18112-0000 at scheduler-fd929918-7057-4fef-923a-ed9d6fd355be@127.0.1.1:44005 I0909 08:02:35.662017 18126 sched.cpp:407] Framework registered with 20140909-080235-16842879-44005-18112-0000 I0909 08:02:35.662039 18126 sched.cpp:421] Scheduler::registered took 9070ns I0909 08:02:35.662119 18126 hierarchical_allocator_process.hpp:329] Added framework 20140909-080235-16842879-44005-18112-0000 I0909 08:02:35.662130 18126 hierarchical_allocator_process.hpp:697] No resources available to allocate! I0909 08:02:35.662135 18126 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 5558ns I0909 08:02:35.672230 18130 leveldb.cpp:343] Persisting action (314 bytes) to leveldb took 13.567526ms I0909 08:02:35.672268 18130 replica.cpp:676] Persisted action at 3 I0909 08:02:35.672483 18130 replica.cpp:655] Replica received learned notice for position 3 I0909 08:02:35.677322 18132 slave.cpp:980] Will retry registration in 14.890338ms if necessary I0909 08:02:35.677399 18132 master.cpp:2831] Ignoring register slave message from slave(64)@127.0.1.1:44005 (precise) as admission is already in progress I0909 08:02:35.680881 18130 leveldb.cpp:343] Persisting action (316 bytes) to leveldb took 8.376798ms I0909 08:02:35.680908 18130 replica.cpp:676] Persisted action at 3 I0909 08:02:35.680917 18130 replica.cpp:661] Replica learned APPEND action at position 3 I0909 08:02:35.681252 18130 registrar.cpp:479] Successfully updated 'registry' I0909 08:02:35.681330 18130 log.cpp:699] Attempting to truncate the log to 3 I0909 08:02:35.681385 18130 master.cpp:2883] Registered slave 20140909-080235-16842879-44005-18112-0 at slave(64)@127.0.1.1:44005 (precise) I0909 08:02:35.681399 18130 master.cpp:4126] Adding slave 20140909-080235-16842879-44005-18112-0 at slave(64)@127.0.1.1:44005 (precise) with cpus(*):3; mem(*):1024; disk(*):25116; ports(*):[31000-32000] I0909 08:02:35.681504 18130 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 I0909 08:02:35.681570 18130 slave.cpp:763] Registered with master master@127.0.1.1:44005; given slave ID 20140909-080235-16842879-44005-18112-0 I0909 08:02:35.681689 18130 slave.cpp:2329] Received ping from slave-observer(50)@127.0.1.1:44005 I0909 08:02:35.681753 18130 hierarchical_allocator_process.hpp:442] Added slave 20140909-080235-16842879-44005-18112-0 (precise) with cpus(*):3; mem(*):1024; disk(*):25116; ports(*):[31000-32000] (and cpus(*):3; mem(*):1024; disk(*):25116; ports(*):[31000-32000] available) I0909 08:02:35.681808 18130 hierarchical_allocator_process.hpp:734] Offering cpus(*):3; mem(*):1024; disk(*):25116; ports(*):[31000-32000] on slave 20140909-080235-16842879-44005-18112-0 to framework 20140909-080235-16842879-44005-18112-0000 I0909 08:02:35.681892 18130 hierarchical_allocator_process.hpp:679] Performed allocation for slave 20140909-080235-16842879-44005-18112-0 in 109580ns I0909 08:02:35.681968 18130 master.hpp:861] Adding offer 20140909-080235-16842879-44005-18112-0 with resources cpus(*):3; mem(*):1024; disk(*):25116; ports(*):[31000-32000] on slave 20140909-080235-16842879-44005-18112-0 (precise) I0909 08:02:35.682014 18130 master.cpp:3600] Sending 1 offers to framework 20140909-080235-16842879-44005-18112-0000 I0909 08:02:35.682443 18130 sched.cpp:544] Scheduler::resourceOffers took 254258ns I0909 08:02:35.682633 18130 master.hpp:871] Removing offer 20140909-080235-16842879-44005-18112-0 with resources cpus(*):3; mem(*):1024; disk(*):25116; ports(*):[31000-32000] on slave 20140909-080235-16842879-44005-18112-0 (precise) I0909 08:02:35.682684 18130 master.cpp:2201] Processing reply for offers: [ 20140909-080235-16842879-44005-18112-0 ] on slave 20140909-080235-16842879-44005-18112-0 at slave(64)@127.0.1.1:44005 (precise) for framework 20140909-080235-16842879-44005-18112-0000 I0909 08:02:35.682708 18130 master.cpp:2284] Authorizing framework principal 'test-principal' to launch task 0 as user 'jenkins' I0909 08:02:35.682971 18130 replica.cpp:508] Replica received write request for position 4 I0909 08:02:35.683132 18132 master.hpp:833] Adding task 0 with resources cpus(*):2; mem(*):512 on slave 20140909-080235-16842879-44005-18112-0 (precise) I0909 08:02:35.683159 18132 master.cpp:2350] Launching task 0 of framework 20140909-080235-16842879-44005-18112-0000 with resources cpus(*):2; mem(*):512 on slave 20140909-080235-16842879-44005-18112-0 at slave(64)@127.0.1.1:44005 (precise) I0909 08:02:35.683363 18132 slave.cpp:1011] Got assigned task 0 for framework 20140909-080235-16842879-44005-18112-0000 I0909 08:02:35.683580 18132 slave.cpp:1121] Launching task 0 for framework 20140909-080235-16842879-44005-18112-0000 I0909 08:02:35.684833 18133 hierarchical_allocator_process.hpp:563] Recovered cpus(*):1; mem(*):512; disk(*):25116; ports(*):[31000-32000] (total allocatable: cpus(*):1; mem(*):512; disk(*):25116; ports(*):[31000-32000]) on slave 20140909-080235-16842879-44005-18112-0 from framework 20140909-080235-16842879-44005-18112-0000 I0909 08:02:35.684864 18133 hierarchical_allocator_process.hpp:599] Framework 20140909-080235-16842879-44005-18112-0000 filtered slave 20140909-080235-16842879-44005-18112-0 for 5secs I0909 08:02:35.686401 18132 exec.cpp:132] Version: 0.21.0 I0909 08:02:35.686848 18128 exec.cpp:182] Executor started at: executor(8)@127.0.1.1:44005 with pid 18112 I0909 08:02:35.687095 18132 slave.cpp:1231] Queuing task '0' for executor executor-1 of framework '20140909-080235-16842879-44005-18112-0000 I0909 08:02:35.687302 18132 slave.cpp:552] Successfully attached file '/tmp/AllocatorTest_0_FrameworkExited_xV9Mk4/slaves/20140909-080235-16842879-44005-18112-0/frameworks/20140909-080235-16842879-44005-18112-0000/executors/executor-1/runs/c4458e43... | 1 |
| 121 | MESOS-1784 | Design the semantics for updating FrameworkInfo | Currently, there is no easy way for frameworks to update their FrameworkInfo., resulting in issues like MESOS-703 and MESOS-1218. This ticket captures the design for doing FrameworkInfo update without having to roll masters/slaves/tasks/executors. | 3 |
| 122 | MESOS-1790 | Add "chown" option to CommandInfo.URI | Mesos fetcher always chown()s the extracted executor URIs as the executor user but sometimes this is not desirable, e.g., "setuid" bit gets lost during chown() if slave/fetcher is running as root. It would be nice to give frameworks the ability to skip the chown. | 2 |
| 123 | MESOS-1799 | Reconciliation can send out-of-order updates. | When a slave re-registers with the master, it currently sends the latest task state for all tasks that are not both terminal and acknowledged. However, reconciliation assumes that we always have the latest unacknowledged state of the task represented in the master. As a result, out-of-order updates are possible, e.g. (1) Slave has task T in TASK_FINISHED, with unacknowledged updates: [TASK_RUNNING, TASK_FINISHED]. (2) Master fails over. (3) New master re-registers the slave with T in TASK_FINISHED. (4) Reconciliation request arrives, master sends TASK_FINISHED. (5) Slave sends TASK_RUNNING to master, master sends TASK_RUNNING. I think the fix here is to preserve the task state invariants in the master, namely, that the master has the latest unacknowledged state of the task. This means when the slave re-registers, it should instead send the latest acknowledged state of each task. | 3 |
| 124 | MESOS-1807 | Disallow executors with cpu only or memory only resources | Currently master allows executors to be launched with either only cpus or only memory but we shouldn't allow that. This is because executor is an actual unix process that is launched by the slave. If an executor doesn't specify cpus, what should do the cpu limits be for that executor when there are no tasks running on it? If no cpu limits are set then it might starve other executors/tasks on the slave violating isolation guarantees. Same goes with memory. Moreover, the current containerizer/isolator code will throw failures when using such an executor, e.g., when the last task on the executor finishes and Containerizer::update() is called with 0 cpus or 0 mem. | 3 |
| 125 | MESOS-1808 | Expose RTT in container stats | As we expose the bandwidth, so we should expose the RTT as a measure of latency each container is experiencing. We can use {{ss}} to get the per-socket statistics and filter and aggregate accordingly to get a measure of RTT. | 3 |
| 126 | MESOS-1811 | Reconcile disconnected/deactivated semantics in the master code | Currently the master code treats a deactivated and disconnected slave similarly, by setting 'disconnected' variable in the slave struct. This causes us to disconnect() a slave in cases where we really only want to deactivate() the slave (e.g., authentication). It would be nice to differentiate these semantics by adding a new variable "active" in the Slave struct. We might want to do the same with the Framework struct for consistency. | 3 |
| 127 | MESOS-1813 | Fail fast in example frameworks if task goes into unexpected state | Most of the example frameworks launch a bunch of tasks and exit if *all* of them reach FINISHED state. But if there is a bug in the code resulting in TASK_LOST, the framework waits forever. Instead the framework should abort if an un-expected task state is encountered. | 1 |
| 128 | MESOS-1814 | Task attempted to use more offers than requested in example jave and python frameworks | {code} [ RUN ] ExamplesTest.JavaFramework Using temporary directory '/tmp/ExamplesTest_JavaFramework_2PcFCh' Enabling authentication for the framework WARNING: Logging before InitGoogleLogging() is written to STDERR I0917 23:14:35.199069 31510 process.cpp:1771] libprocess is initialized on 127.0.1.1:34609 for 8 cpus I0917 23:14:35.199794 31510 logging.cpp:177] Logging to STDERR I0917 23:14:35.225342 31510 leveldb.cpp:176] Opened db in 22.197149ms I0917 23:14:35.231133 31510 leveldb.cpp:183] Compacted db in 5.601897ms I0917 23:14:35.231498 31510 leveldb.cpp:198] Created db iterator in 215441ns I0917 23:14:35.231608 31510 leveldb.cpp:204] Seeked to beginning of db in 11488ns I0917 23:14:35.231722 31510 leveldb.cpp:273] Iterated through 0 keys in the db in 14016ns I0917 23:14:35.231917 31510 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0917 23:14:35.233129 31526 recover.cpp:425] Starting replica recovery I0917 23:14:35.233614 31526 recover.cpp:451] Replica is in EMPTY status I0917 23:14:35.234994 31526 replica.cpp:638] Replica in EMPTY status received a broadcasted recover request I0917 23:14:35.240116 31519 recover.cpp:188] Received a recover response from a replica in EMPTY status I0917 23:14:35.240782 31519 recover.cpp:542] Updating replica status to STARTING I0917 23:14:35.242846 31524 master.cpp:286] Master 20140917-231435-16842879-34609-31503 (saucy) started on 127.0.1.1:34609 I0917 23:14:35.243191 31524 master.cpp:332] Master only allowing authenticated frameworks to register I0917 23:14:35.243288 31524 master.cpp:339] Master allowing unauthenticated slaves to register I0917 23:14:35.243399 31524 credentials.hpp:36] Loading credentials for authentication from '/tmp/ExamplesTest_JavaFramework_2PcFCh/credentials' W0917 23:14:35.243588 31524 credentials.hpp:51] Permissions on credentials file '/tmp/ExamplesTest_JavaFramework_2PcFCh/credentials' are too open. It is recommended that your credentials file is NOT accessible by others. I0917 23:14:35.243846 31524 master.cpp:366] Authorization enabled I0917 23:14:35.244882 31520 hierarchical_allocator_process.hpp:299] Initializing hierarchical allocator process with master : master@127.0.1.1:34609 I0917 23:14:35.245224 31520 master.cpp:120] No whitelist given. Advertising offers for all slaves I0917 23:14:35.246934 31524 master.cpp:1211] The newly elected leader is master@127.0.1.1:34609 with id 20140917-231435-16842879-34609-31503 I0917 23:14:35.247234 31524 master.cpp:1224] Elected as the leading master! I0917 23:14:35.247336 31524 master.cpp:1042] Recovering from registrar I0917 23:14:35.247542 31526 registrar.cpp:313] Recovering registrar I0917 23:14:35.250555 31510 containerizer.cpp:89] Using isolation: posix/cpu,posix/mem I0917 23:14:35.252326 31510 containerizer.cpp:89] Using isolation: posix/cpu,posix/mem I0917 23:14:35.252821 31520 slave.cpp:169] Slave started on 1)@127.0.1.1:34609 I0917 23:14:35.253552 31520 slave.cpp:289] Slave resources: cpus(*):1; mem(*):1001; disk(*):24988; ports(*):[31000-32000] I0917 23:14:35.253906 31520 slave.cpp:317] Slave hostname: saucy I0917 23:14:35.254004 31520 slave.cpp:318] Slave checkpoint: true I0917 23:14:35.254818 31520 state.cpp:33] Recovering state from '/tmp/mesos-w8snRW/0/meta' I0917 23:14:35.255106 31519 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 13.99622ms I0917 23:14:35.255235 31519 replica.cpp:320] Persisted replica status to STARTING I0917 23:14:35.255419 31519 recover.cpp:451] Replica is in STARTING status I0917 23:14:35.255834 31519 replica.cpp:638] Replica in STARTING status received a broadcasted recover request I0917 23:14:35.256000 31519 recover.cpp:188] Received a recover response from a replica in STARTING status I0917 23:14:35.256217 31519 recover.cpp:542] Updating replica status to VOTING I0917 23:14:35.256641 31520 status_update_manager.cpp:193] Recovering status update manager I0917 23:14:35.257064 31520 containerizer.cpp:252] Recovering containerizer I0917 23:14:35.257725 31520 slave.cpp:3220] Finished recovery I0917 23:14:35.258463 31520 slave.cpp:600] New master detected at master@127.0.1.1:34609 I0917 23:14:35.258769 31524 status_update_manager.cpp:167] New master detected at master@127.0.1.1:34609 I0917 23:14:35.258885 31520 slave.cpp:636] No credentials provided. Attempting to register without authentication I0917 23:14:35.259024 31520 slave.cpp:647] Detecting new master I0917 23:14:35.259863 31520 slave.cpp:169] Slave started on 2)@127.0.1.1:34609 I0917 23:14:35.260288 31520 slave.cpp:289] Slave resources: cpus(*):1; mem(*):1001; disk(*):24988; ports(*):[31000-32000] I0917 23:14:35.260493 31520 slave.cpp:317] Slave hostname: saucy I0917 23:14:35.260588 31520 slave.cpp:318] Slave checkpoint: true I0917 23:14:35.265127 31510 containerizer.cpp:89] Using isolation: posix/cpu,posix/mem I0917 23:14:35.265877 31519 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 9.536278ms I0917 23:14:35.265983 31519 replica.cpp:320] Persisted replica status to VOTING I0917 23:14:35.266324 31519 recover.cpp:556] Successfully joined the Paxos group I0917 23:14:35.266511 31519 recover.cpp:440] Recover process terminated I0917 23:14:35.266978 31519 log.cpp:656] Attempting to start the writer I0917 23:14:35.268165 31523 replica.cpp:474] Replica received implicit promise request with proposal 1 I0917 23:14:35.269850 31525 slave.cpp:169] Slave started on 3)@127.0.1.1:34609 I0917 23:14:35.270365 31525 slave.cpp:289] Slave resources: cpus(*):1; mem(*):1001; disk(*):24988; ports(*):[31000-32000] I0917 23:14:35.270658 31525 slave.cpp:317] Slave hostname: saucy I0917 23:14:35.270781 31525 slave.cpp:318] Slave checkpoint: true I0917 23:14:35.271332 31525 state.cpp:33] Recovering state from '/tmp/mesos-w8snRW/2/meta' I0917 23:14:35.271580 31522 status_update_manager.cpp:193] Recovering status update manager I0917 23:14:35.271838 31522 containerizer.cpp:252] Recovering containerizer I0917 23:14:35.272238 31525 slave.cpp:3220] Finished recovery I0917 23:14:35.273002 31525 slave.cpp:600] New master detected at master@127.0.1.1:34609 I0917 23:14:35.273252 31521 status_update_manager.cpp:167] New master detected at master@127.0.1.1:34609 I0917 23:14:35.273360 31525 slave.cpp:636] No credentials provided. Attempting to register without authentication I0917 23:14:35.273507 31525 slave.cpp:647] Detecting new master I0917 23:14:35.275413 31525 state.cpp:33] Recovering state from '/tmp/mesos-w8snRW/1/meta' I0917 23:14:35.278506 31523 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 10.232514ms I0917 23:14:35.278712 31523 replica.cpp:342] Persisted promised to 1 I0917 23:14:35.279585 31523 coordinator.cpp:230] Coordinator attemping to fill missing position I0917 23:14:35.280400 31523 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I0917 23:14:35.280900 31526 status_update_manager.cpp:193] Recovering status update manager I0917 23:14:35.281282 31519 containerizer.cpp:252] Recovering containerizer I0917 23:14:35.281615 31520 slave.cpp:3220] Finished recovery I0917 23:14:35.281891 31510 sched.cpp:137] Version: 0.21.0 I0917 23:14:35.282306 31526 sched.cpp:233] New master detected at master@127.0.1.1:34609 I0917 23:14:35.282464 31526 sched.cpp:283] Authenticating with master master@127.0.1.1:34609 I0917 23:14:35.282891 31526 authenticatee.hpp:104] Initializing client SASL I0917 23:14:35.284816 31526 authenticatee.hpp:128] Creating new client SASL connection I0917 23:14:35.285428 31519 master.cpp:873] Dropping 'mesos.internal.AuthenticateMessage' message since not recovered yet I0917 23:14:35.288007 31521 slave.cpp:600] New master detected at master@127.0.1.1:34609 I0917 23:14:35.288399 31521 slave.cpp:636] No credentials provided. Attempting to register without authentication I0917 23:14:35.288535 31521 slave.cpp:647] Detecting new master I0917 23:14:35.288501 31519 status_update_manager.cpp:167] New master detected at master@127.0.1.1:34609 I0917 23:14:35.289625 31523 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 8.997343ms I0917 23:14:35.289784 31523 replica.cpp:676] Persisted action at 0 I0917 23:14:35.292667 31521 replica.cpp:508] Replica received write request for position 0 I0917 23:14:35.293112 31521 leveldb.cpp:438] Reading position from leveldb took 325638ns I0917 23:14:35.301774 31521 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 8.576338ms I0917 23:14:35.301916 31521 replica.cpp:676] Persisted action at 0 I0917 23:14:35.302289 31521 replica.cpp:655] Replica received learned notice for position 0 I0917 23:14:35.310542 31521 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 8.087789ms I0917 23:14:35.310675 31521 replica.cpp:676] Persisted action at 0 I0917 23:14:35.310946 31521 replica.cpp:661] Replica learned NOP action at position 0 I0917 23:14:35.311254 31521 log.cpp:672] Writer started with ending position 0 I0917 23:14:35.311957 31521 leveldb.cpp:438] Reading position from leveldb took 35110ns I0917 23:14:35.320283 31521 registrar.cpp:346] Successfully fetched the registry (0B) I0917 23:14:35.320513 31521 registrar.cpp:422] Attempting to update the 'registry' I0917 23:14:35.322226 31525 log.cpp:680] Attempting to append 118 bytes to the log I0917 23:14:35.322549 31525 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0917 23:14:35.322931 31525 replica.cpp:508] Replica received write request for position 1 I0917 23:14:35.330169 31525 leveldb.cpp:343] Persisting action (135 bytes) to leveldb took 7.133053ms I0917 23:14:35.330340 31525 replica.cpp:676] Persisted action at 1 I0917 23:14:35.330890 31525 replica.cpp:655] Replica received learned notice for position 1 I0917 23:14:35.339218 31525 leveldb.cpp:343] Persisting action (137 bytes) to leveldb took 8.192024ms I0917 23:14:35.339380 31525 replica.cpp:676] Persisted action at 1 I0917 23:14:35.339715 31525 replica.cpp:661] Replica learned APPEND action at position 1 I0917 23:14:35.340615 31525 registrar.cpp:479] Successfully updated 'registry' I0917 23:14:35.340802 31525 registrar.cpp:372] Successfully recovered registrar I0917 23:14:35.341104 31525 log.cpp:699] Attempting to truncate the log to 1 I0917 23:14:35.341351 31525 master.cpp:1069] Recovered 0 slaves from the Registry (82B) ; allowing 10mins for slaves to re-register I0917 23:14:35.341527 31525 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0917 23:14:35.341964 31525 replica.cpp:508] Replica received write request for position 2 I0917 23:14:35.352336 31525 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 10.213086ms I0917 23:14:35.352494 31525 replica.cpp:676] Persisted action at 2 I0917 23:14:35.356258 31523 replica.cpp:655] Replica received learned notice for position 2 I0917 23:14:35.364992 31523 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 8.606522ms I0917 23:14:35.365166 31523 leveldb.cpp:401] Deleting ~1 keys from leveldb took 48378ns I0917 23:14:35.365404 31523 replica.cpp:676] Persisted action at 2 I0917 23:14:35.365537 31523 replica.cpp:661] Replica learned TRUNCATE action at position 2 I0917 23:14:35.568366 31523 slave.cpp:994] Will retry registration in 423.208575ms if necessary I0917 23:14:35.568840 31522 master.cpp:2870] Registering slave at slave(3)@127.0.1.1:34609 (saucy) with id 20140917-231435-16842879-34609-31503-0 I0917 23:14:35.569422 31522 registrar.cpp:422] Attempting to update the 'registry' I0917 23:14:35.572013 31522 log.cpp:680] Attempting to append 289 bytes to the log I0917 23:14:35.572273 31519 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I0917 23:14:35.572816 31519 replica.cpp:508] Replica received write request for position 3 I0917 23:14:35.579784 31519 leveldb.cpp:343] Persisting action (308 bytes) to leveldb took 6.809365ms I0917 23:14:35.579907 31519 replica.cpp:676] Persisted action at 3 I0917 23:14:35.580512 31519 replica.cpp:655] Replica received learned notice for position 3 I0917 23:14:35.588748 31519 leveldb.cpp:343] Persisting action (310 bytes) to leveldb took 8.112519ms I0917 23:14:35.588888 31519 replica.cpp:676] Persisted action at 3 I0917 23:14:35.588985 31519 replica.cpp:661] Replica learned APPEND action at position 3 I0917 23:14:35.589754 31519 registrar.cpp:479] Successfully updated 'registry' I0917 23:14:35.590070 31519 master.cpp:2910] Registered slave 20140917-231435-16842879-34609-31503-0 at slave(3)@127.0.1.1:34609 (saucy) I0917 23:14:35.590255 31519 master.cpp:4118] Adding slave 20140917-231435-16842879-34609-31503-0 at slave(3)@127.0.1.1:34609 (saucy) with cpus(*):1; mem(*):1001; disk(*):24988; ports(*):[31000-32000] I0917 23:14:35.590831 31519 slave.cpp:765] Registered with master master@127.0.1.1:34609; given slave ID 20140917-231435-16842879-34609-31503-0 I0917 23:14:35.589913 31523 log.cpp:699] Attempting to truncate the log to 3 I0917 23:14:35.591414 31523 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 I0917 23:14:35.591815 31523 replica.cpp:508] Replica received write request for position 4 I0917 23:14:35.591117 31521 hierarchical_allocator_process.hpp:442] Added slave 20140917-231435-16842879-34609-31503-0 (saucy) with cpus(*):1; mem(*):1001; disk(*):24988; ports(*):[31000-32000] (and cpus(*):1; mem(*):1001; disk(*):24988; ports(*):[31000-32000] available) I0917 23:14:35.592293 31521 hierarchical_allocator_process.hpp:679] Performed allocation for slave 20140917-231435-16842879-34609-31503-0 in 64364ns I0917 23:14:35.592953 31519 slave.cpp:778] Checkpointing SlaveInfo to '/tmp/mesos-w8snRW/2/meta/slaves/20140917-231435-16842879-34609-31503-0/slave.info' I0917 23:14:35.593475 31519 slave.cpp:2347] Received ping from slave-observer(1)@127.0.1.1:34609 I0917 23:14:35.601356 31523 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 9.420461ms I0917 23:14:35.601539 31523 replica.cpp:676] Persisted action at 4 I0917 23:14:35.602325 31523 replica.cpp:655] Replica received learned notice for position 4 I0917 23:14:35.610779 31523 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 8.34398ms I0917 23:14:35.611114 31523 leveldb.cpp:401] Deleting ~2 keys from leveldb took 66521ns I0917 23:14:35.611554 31523 replica.cpp:676] Persisted action at 4 I0917 23:14:35.611690 31523 replica.cpp:661] Replica learned TRUNCATE action at position 4 I0917 23:14:36.033941 31523 slave.cpp:994] Will retry registration in 322.705631ms if necessary I0917 23:14:36.034276 31521 master.cpp:2870] Registering slave at slave(1)@127.0.1.1:34609 (saucy) with id 20140917-231435-16842879-34609-31503-1 I0917 23:14:36.034536 31521 registrar.cpp:422] Attempting to update the 'registry' I0917 23:14:36.035889 31521 log.cpp:680] Attempting to append 454 bytes to the log I0917 23:14:36.036099 31524 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 5 I0917 23:14:36.036416 31524 replica.cpp:508] Replica received write request for position 5 I0917 23:14:36.046672 31524 leveldb.cpp:343] Persisting action (473 bytes) to leveldb took 10.160627ms I0917 23:14:36.047035 31524 replica.cpp:676] Persisted action at 5 I0917 23:14:36.047613 31524 replica.cpp:655] Replica received learned notice for position 5 I0917 23:14:36.053006 31524 leveldb.cpp:343] Persisting action (475 bytes) to leveldb took 5.180742ms I0917 23:14:36.053246 31524 replica.cpp:676] Persisted action at 5 I0917 23:14:36.053678 31524 replica.cpp:661] Replica learned APPEND action at position 5 I0917 23:14:36.060384 31524 registrar.cpp:479] Successfully updated 'registry' I0917 23:14:36.061328 31524 master.cpp:2910] Registered slave 20140917-231435-16842879-34609-31503-1 at slave(1)@127.0.1.1:34609 (saucy) I0917 23:14:36.061537 31524 master.cpp:4118] Adding slave 20140917-231435-16842879-34609-31503-1 at slave(1)@127.0.1.1:34609 (saucy) with cpus(*):1; mem(*):1001; disk(*):24988; ports(*):[31000-32000] I0917 23:14:36.061982 31524 slave.cpp:765] Registered with master master@127.0.1.1:34609; given slave ID 20140917-231435-16842879-34609-31503-1 I0917 23:14:36.062891 31524 slave.cpp:778] Checkpointing SlaveInfo to '/tmp/mesos-w8snRW/0/meta/slaves/20140917-231435-16842879-34609-31503-1/slave.info' I0917 23:14:36.061050 31525 log.cpp:699] Attempting to truncate the log to 5 I0917 23:14:36.063244 31525 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 6 I0917 23:14:36.063746 31525 replica.cpp:508] Replica received write request for position 6 I0917 23:14:36.062386 31520 hierarchical_allocator_process.hpp:442] Added slave 20140917-231435-16842879-34609-31503-1 (saucy) with cpus(*):1; mem(*):1001; disk(*):24988; ports(*):[31000-32000] (and cpus(*):1; mem(*):1001; disk(*):24988; ports(*):[31000-32000] available) I0917 23:14:36.064352 31520 hierarchical_allocator_process.hpp:679] Performed allocation for slave 20140917-231435-16842879-34609-31503-1 in 35730ns I0917 23:14:36.065166 31524 slave.cpp:2347] Received ping from slave-observer(2)@127.0.1.1:34609 I0917 23:14:36.070137 31525 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 6.242192ms I0917 23:14:36.070355 31525 replica.cpp:676] Persisted action at 6 I0917 23:14:36.071005 31525 replica.cpp:655] Replica received learned notice for position 6 I0917 23:14:36.076560 31525 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 5.368532ms I0917 23:14:36.077137 31525 leveldb.cpp:401] Deleting ~2 keys from leveldb took 371245ns I0917 23:14:36.077241 31525 replica.cpp:676] Persisted action at 6 I0917 23:14:36.077345 31525 replica.cpp:661] Replica learned TRUNCATE action at position 6 I0917 23:14:36.141270 31522 slave.cpp:994] Will retry registration in 1.857205901secs if necessary I0917 23:14:36.141644 31522 master.cpp:2870] Registering slave at slave(2)@127.0.1.1:34609 (saucy) with id 20140917-231435-16842879-34609-31503-2 I0917 23:14:36.141930 31522 registrar.cpp:422] Attempting to update the 'registry' I0917 23:14:36.143316 31521 log.cpp:680] Attempting to append 619 bytes to the log I0917 23:14:36.143646 31521 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 7 I0917 23:14:36.143954 31521 replica.cpp:508] Replica received write request for position 7 I0917 23:14:36.148875 31521 leveldb.cpp:343] Persisting action (638 bytes) to leveldb took 4.787834ms I0917 23:14:36.149085 31521 replica.cpp:676] Persisted action at 7 I0917 23:14:36.149673 31521 replica.cpp:655] Replica received learned notice for position 7 I0917 23:14:36.155232 31521 leveldb.cpp:343] Persisting action (640 bytes) to leveldb took 5.472209ms I0917 23:14:36.155522 31521 replica.cpp:676] Persisted action at 7 I0917 23:14:36.155936 31521 replica.cpp:661] Replica learned APPEND action at position 7 I0917 23:14:36.156481 31521 registrar.cpp:479] Successfully updated 'registry' I0917 23:14:36.156663 31526 log.cpp:699] Attempting to truncate the log to 7 I0917 23:14:36.156813 31526 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 8 I0917 23:14:36.157155 31526 replica.cpp:508] Replica received write request for position 8 I0917 23:14:36.157510 31520 master.cpp:2910] Registered slave 20140917-231435-16842879-34609-31503-2 at slave(2)@127.0.1.1:34609 (saucy) I0917 23:14:36.157645 31520 master.cpp:4118] Adding slave 20140917-231435-16842879-34609-31503-2 at slave(2)@127.0.1.1:34609 (saucy) with cpus(*):1; mem(*):1001; disk(*):24988; ports(*):[31000-32000] I0917 23:14:36.157928 31520 slave.cpp:765] Registered with master master@127.0.1.1:34609; given slave ID 20140917-231435-16842879-34609-31503-2 I0917 23:14:36.158304 31520 slave.cpp:778] Checkpointing SlaveInfo to '/tmp/mesos-w8snRW/1/meta/slaves/20140917-231435-16842879-34609-31503-2/slave.info' I0... | 2 |
| 129 | MESOS-1815 | Create a guide to becoming a committer | We have a committer's guide, but the process by which one becomes a committer is unclear. We should set some guidelines and a process by which we can grow contributors into committers. | 3 |
| 130 | MESOS-1817 | Completed tasks remains in TASK_RUNNING when framework is disconnected | We have run into a problem that cause tasks which completes, when a framework is disconnected and has a fail-over time, to remain in a running state even though the tasks actually finishes. This hogs the cluster and gives users a inconsistent view of the cluster state. Going to the slave, the task is finished. Going to the master, the task is still in a non-terminal state. When the scheduler reattaches or the failover timeout expires, the tasks finishes correctly. The current workflow of this scheduler has a long fail-over timeout, but may on the other hand never reattach. Here is a test framework we have been able to reproduce the issue with: https://gist.github.com/nqn/9b9b1de9123a6e836f54 It launches many short-lived tasks (1 second sleep) and when killing the framework instance, the master reports the tasks as running even after several minutes: http://cl.ly/image/2R3719461e0t/Screen%20Shot%202014-09-10%20at%203.19.39%20PM.png When clicking on one of the slaves where, for example, task 49 runs; the slave knows that it completed: http://cl.ly/image/2P410L3m1O1N/Screen%20Shot%202014-09-10%20at%203.21.29%20PM.png Here is the log of a mesos-local instance where I reproduced it: https://gist.github.com/nqn/f7ee20601199d70787c0 (Here task 10 to 19 are stuck in running state). There is a lot of output, so here is a filtered log for task 10: https://gist.github.com/nqn/a53e5ea05c5e41cd5a7d The problem turn out to be an issue with the ack-cycle of status updates: If the framework disconnects (with a failover timeout set), the status update manage on the slaves will keep trying to send the front of status update stream to the master (which in turn forwards it to the framework). If the first status update after the disconnect is terminal, things work out fine; the master pick the terminal state up, removes the task and release the resources. If, on the other hand, one non-terminal status is in the stream. The master will never know that the task finished (or failed) before the framework reconnects. During a discussion on the dev mailing list (http://mail-archives.apache.org/mod_mbox/mesos-dev/201409.mbox/%3cCADKthhAVR5mrq1s9HXw1BB_XFALXWWxjutp7MV4y3wP-Bh=aWg@mail.gmail.com%3e) we enumerated a couple of options to solve this problem. First off, having two ack-cycles: one between masters and slaves and one between masters and frameworks, would be ideal. We would be able to replay the statuses in order while keeping the master state current. However, this requires us to persist the master state in a replicated storage. As a first pass, we can make sure that the tasks caught in a running state doesn't hog the cluster when completed and the framework being disconnected. Here is a proof-of-concept to work out of: https://github.com/nqn/mesos/tree/niklas/status-update-disconnect/ A new (optional) field have been added to the internal status update message: https://github.com/nqn/mesos/blob/niklas/status-update-disconnect/src/messages/messages.proto#L68 Which makes it possible for the status update manager to set the field, if the latest status was terminal: https://github.com/nqn/mesos/blob/niklas/status-update-disconnect/src/slave/status_update_manager.cpp#L501 I added a test which should high-light the issue as well: https://github.com/nqn/mesos/blob/niklas/status-update-disconnect/src/tests/fault_tolerance_tests.cpp#L2478 I would love some input on the approach before moving on. There are rough edges in the PoC which (of course) should be addressed before bringing it for up review. | 2 |
| 131 | MESOS-1830 | Expose master stats differentiating between master-generated and slave-generated LOST tasks | The master exports a monotonically-increasing counter of tasks transitioned to TASK_LOST. This loses fidelity of the source of the lost task. A first step in exposing the source of lost tasks might be to just differentiate between TASK_LOST transitions initiated by the master vs the slave (and maybe bad input from the scheduler). | 5 |
| 132 | MESOS-1844 | AllocatorTest/0.SlaveLost is flaky | {code} [ RUN ] AllocatorTest/0.SlaveLost Using temporary directory '/tmp/AllocatorTest_0_SlaveLost_Z2oazw' I0929 16:58:29.484141 3486 leveldb.cpp:176] Opened db in 604109ns I0929 16:58:29.484629 3486 leveldb.cpp:183] Compacted db in 172697ns I0929 16:58:29.484912 3486 leveldb.cpp:198] Created db iterator in 6429ns I0929 16:58:29.485133 3486 leveldb.cpp:204] Seeked to beginning of db in 1618ns I0929 16:58:29.485337 3486 leveldb.cpp:273] Iterated through 0 keys in the db in 752ns I0929 16:58:29.485595 3486 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0929 16:58:29.486017 3500 recover.cpp:425] Starting replica recovery I0929 16:58:29.486304 3500 recover.cpp:451] Replica is in EMPTY status I0929 16:58:29.486793 3500 replica.cpp:638] Replica in EMPTY status received a broadcasted recover request I0929 16:58:29.487205 3500 recover.cpp:188] Received a recover response from a replica in EMPTY status I0929 16:58:29.487540 3500 recover.cpp:542] Updating replica status to STARTING I0929 16:58:29.487911 3500 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 36629ns I0929 16:58:29.488173 3500 replica.cpp:320] Persisted replica status to STARTING I0929 16:58:29.488438 3500 recover.cpp:451] Replica is in STARTING status I0929 16:58:29.488891 3500 replica.cpp:638] Replica in STARTING status received a broadcasted recover request I0929 16:58:29.489187 3500 recover.cpp:188] Received a recover response from a replica in STARTING status I0929 16:58:29.489516 3500 recover.cpp:542] Updating replica status to VOTING I0929 16:58:29.489887 3502 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 32099ns I0929 16:58:29.490124 3502 replica.cpp:320] Persisted replica status to VOTING I0929 16:58:29.490381 3500 recover.cpp:556] Successfully joined the Paxos group I0929 16:58:29.490713 3500 recover.cpp:440] Recover process terminated I0929 16:58:29.493401 3506 master.cpp:312] Master 20140929-165829-2759502016-55618-3486 (fedora-20) started on 192.168.122.164:55618 I0929 16:58:29.493700 3506 master.cpp:358] Master only allowing authenticated frameworks to register I0929 16:58:29.493921 3506 master.cpp:363] Master only allowing authenticated slaves to register I0929 16:58:29.494123 3506 credentials.hpp:36] Loading credentials for authentication from '/tmp/AllocatorTest_0_SlaveLost_Z2oazw/credentials' I0929 16:58:29.494500 3506 master.cpp:392] Authorization enabled I0929 16:58:29.495249 3506 master.cpp:120] No whitelist given. Advertising offers for all slaves I0929 16:58:29.495728 3502 hierarchical_allocator_process.hpp:299] Initializing hierarchical allocator process with master : master@192.168.122.164:55618 I0929 16:58:29.496196 3506 master.cpp:1241] The newly elected leader is master@192.168.122.164:55618 with id 20140929-165829-2759502016-55618-3486 I0929 16:58:29.496469 3506 master.cpp:1254] Elected as the leading master! I0929 16:58:29.496713 3506 master.cpp:1072] Recovering from registrar I0929 16:58:29.497020 3506 registrar.cpp:312] Recovering registrar I0929 16:58:29.497486 3506 log.cpp:656] Attempting to start the writer I0929 16:58:29.498105 3506 replica.cpp:474] Replica received implicit promise request with proposal 1 I0929 16:58:29.498373 3506 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 27145ns I0929 16:58:29.498605 3506 replica.cpp:342] Persisted promised to 1 I0929 16:58:29.500880 3500 coordinator.cpp:230] Coordinator attemping to fill missing position I0929 16:58:29.501404 3500 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I0929 16:58:29.501687 3500 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 57971ns I0929 16:58:29.501935 3500 replica.cpp:676] Persisted action at 0 I0929 16:58:29.504905 3507 replica.cpp:508] Replica received write request for position 0 I0929 16:58:29.505130 3507 leveldb.cpp:438] Reading position from leveldb took 18418ns I0929 16:58:29.505377 3507 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 19998ns I0929 16:58:29.505571 3507 replica.cpp:676] Persisted action at 0 I0929 16:58:29.505957 3507 replica.cpp:655] Replica received learned notice for position 0 I0929 16:58:29.506186 3507 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 21648ns I0929 16:58:29.506433 3507 replica.cpp:676] Persisted action at 0 I0929 16:58:29.506767 3507 replica.cpp:661] Replica learned NOP action at position 0 I0929 16:58:29.507199 3507 log.cpp:672] Writer started with ending position 0 I0929 16:58:29.507730 3507 leveldb.cpp:438] Reading position from leveldb took 11532ns I0929 16:58:29.508915 3507 registrar.cpp:345] Successfully fetched the registry (0B) I0929 16:58:29.509230 3507 registrar.cpp:421] Attempting to update the 'registry' I0929 16:58:29.510516 3500 log.cpp:680] Attempting to append 130 bytes to the log I0929 16:58:29.510949 3500 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0929 16:58:29.511363 3500 replica.cpp:508] Replica received write request for position 1 I0929 16:58:29.511697 3500 leveldb.cpp:343] Persisting action (149 bytes) to leveldb took 66530ns I0929 16:58:29.512039 3500 replica.cpp:676] Persisted action at 1 I0929 16:58:29.512460 3500 replica.cpp:655] Replica received learned notice for position 1 I0929 16:58:29.512778 3500 leveldb.cpp:343] Persisting action (151 bytes) to leveldb took 24121ns I0929 16:58:29.513013 3500 replica.cpp:676] Persisted action at 1 I0929 16:58:29.513239 3500 replica.cpp:661] Replica learned APPEND action at position 1 I0929 16:58:29.513674 3500 log.cpp:699] Attempting to truncate the log to 1 I0929 16:58:29.513954 3500 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0929 16:58:29.514385 3500 replica.cpp:508] Replica received write request for position 2 I0929 16:58:29.514680 3500 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 65014ns I0929 16:58:29.514991 3500 replica.cpp:676] Persisted action at 2 I0929 16:58:29.516978 3501 replica.cpp:655] Replica received learned notice for position 2 I0929 16:58:29.517319 3501 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 24103ns I0929 16:58:29.517546 3501 leveldb.cpp:401] Deleting ~1 keys from leveldb took 16533ns I0929 16:58:29.517801 3501 replica.cpp:676] Persisted action at 2 I0929 16:58:29.518039 3501 replica.cpp:661] Replica learned TRUNCATE action at position 2 I0929 16:58:29.518539 3507 registrar.cpp:478] Successfully updated 'registry' I0929 16:58:29.518885 3507 registrar.cpp:371] Successfully recovered registrar I0929 16:58:29.519201 3507 master.cpp:1099] Recovered 0 slaves from the Registry (94B) ; allowing 10mins for slaves to re-register I0929 16:58:29.533073 3505 slave.cpp:169] Slave started on 57)@192.168.122.164:55618 I0929 16:58:29.533500 3505 credentials.hpp:84] Loading credential for authentication from '/tmp/AllocatorTest_0_SlaveLost_xdXHfg/credential' I0929 16:58:29.533834 3505 slave.cpp:276] Slave using credential for: test-principal I0929 16:58:29.534168 3505 slave.cpp:289] Slave resources: cpus(*):2; mem(*):1024; disk(*):752; ports(*):[31000-32000] I0929 16:58:29.534751 3505 slave.cpp:317] Slave hostname: fedora-20 I0929 16:58:29.534965 3505 slave.cpp:318] Slave checkpoint: false I0929 16:58:29.535557 3505 state.cpp:33] Recovering state from '/tmp/AllocatorTest_0_SlaveLost_xdXHfg/meta' I0929 16:58:29.535951 3505 status_update_manager.cpp:193] Recovering status update manager I0929 16:58:29.536290 3505 slave.cpp:3271] Finished recovery I0929 16:58:29.536782 3505 slave.cpp:598] New master detected at master@192.168.122.164:55618 I0929 16:58:29.537122 3505 slave.cpp:672] Authenticating with master master@192.168.122.164:55618 I0929 16:58:29.537492 3505 slave.cpp:645] Detecting new master I0929 16:58:29.537294 3506 status_update_manager.cpp:167] New master detected at master@192.168.122.164:55618 I0929 16:58:29.537642 3507 authenticatee.hpp:128] Creating new client SASL connection I0929 16:58:29.538769 3502 master.cpp:3737] Authenticating slave(57)@192.168.122.164:55618 I0929 16:58:29.539091 3502 authenticator.hpp:156] Creating new server SASL connection I0929 16:58:29.539710 3503 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0929 16:58:29.539943 3503 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0929 16:58:29.540206 3502 authenticator.hpp:262] Received SASL authentication start I0929 16:58:29.540457 3502 authenticator.hpp:384] Authentication requires more steps I0929 16:58:29.540757 3502 authenticatee.hpp:265] Received SASL authentication step I0929 16:58:29.541121 3502 authenticator.hpp:290] Received SASL authentication step I0929 16:58:29.541368 3502 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'fedora-20' server FQDN: 'fedora-20' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0929 16:58:29.541599 3502 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0929 16:58:29.541874 3502 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0929 16:58:29.542129 3502 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'fedora-20' server FQDN: 'fedora-20' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0929 16:58:29.542333 3502 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0929 16:58:29.542553 3502 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0929 16:58:29.542785 3502 authenticator.hpp:376] Authentication success I0929 16:58:29.543047 3502 authenticatee.hpp:305] Authentication success I0929 16:58:29.543381 3502 slave.cpp:729] Successfully authenticated with master master@192.168.122.164:55618 I0929 16:58:29.543707 3502 slave.cpp:992] Will retry registration in 11.795692ms if necessary I0929 16:58:29.543179 3503 master.cpp:3777] Successfully authenticated principal 'test-principal' at slave(57)@192.168.122.164:55618 I0929 16:58:29.544255 3503 master.cpp:2930] Registering slave at slave(57)@192.168.122.164:55618 (fedora-20) with id 20140929-165829-2759502016-55618-3486-0 I0929 16:58:29.544587 3503 registrar.cpp:421] Attempting to update the 'registry' I0929 16:58:29.545816 3500 log.cpp:680] Attempting to append 299 bytes to the log I0929 16:58:29.546267 3500 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I0929 16:58:29.546749 3500 replica.cpp:508] Replica received write request for position 3 I0929 16:58:29.547030 3500 leveldb.cpp:343] Persisting action (318 bytes) to leveldb took 31759ns I0929 16:58:29.547236 3500 replica.cpp:676] Persisted action at 3 I0929 16:58:29.548902 3506 replica.cpp:655] Replica received learned notice for position 3 I0929 16:58:29.549139 3506 leveldb.cpp:343] Persisting action (320 bytes) to leveldb took 25595ns I0929 16:58:29.549343 3506 replica.cpp:676] Persisted action at 3 I0929 16:58:29.549607 3506 replica.cpp:661] Replica learned APPEND action at position 3 I0929 16:58:29.550081 3506 log.cpp:699] Attempting to truncate the log to 3 I0929 16:58:29.550497 3506 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 I0929 16:58:29.550943 3506 replica.cpp:508] Replica received write request for position 4 I0929 16:58:29.551198 3506 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 20852ns I0929 16:58:29.551409 3506 replica.cpp:676] Persisted action at 4 I0929 16:58:29.551795 3506 replica.cpp:655] Replica received learned notice for position 4 I0929 16:58:29.552094 3506 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 22182ns I0929 16:58:29.552320 3506 leveldb.cpp:401] Deleting ~2 keys from leveldb took 18503ns I0929 16:58:29.552525 3506 replica.cpp:676] Persisted action at 4 I0929 16:58:29.552781 3506 replica.cpp:661] Replica learned TRUNCATE action at position 4 I0929 16:58:29.550289 3503 registrar.cpp:478] Successfully updated 'registry' I0929 16:58:29.553553 3503 master.cpp:2970] Registered slave 20140929-165829-2759502016-55618-3486-0 at slave(57)@192.168.122.164:55618 (fedora-20) I0929 16:58:29.553807 3503 master.cpp:4180] Adding slave 20140929-165829-2759502016-55618-3486-0 at slave(57)@192.168.122.164:55618 (fedora-20) with cpus(*):2; mem(*):1024; disk(*):752; ports(*):[31000-32000] I0929 16:58:29.554152 3503 slave.cpp:763] Registered with master master@192.168.122.164:55618; given slave ID 20140929-165829-2759502016-55618-3486-0 I0929 16:58:29.554455 3503 slave.cpp:2345] Received ping from slave-observer(56)@192.168.122.164:55618 I0929 16:58:29.554707 3504 hierarchical_allocator_process.hpp:442] Added slave 20140929-165829-2759502016-55618-3486-0 (fedora-20) with cpus(*):2; mem(*):1024; disk(*):752; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):752; ports(*):[31000-32000] available) I0929 16:58:29.555064 3504 hierarchical_allocator_process.hpp:679] Performed allocation for slave 20140929-165829-2759502016-55618-3486-0 in 13111ns I0929 16:58:29.558220 3486 sched.cpp:137] Version: 0.21.0 I0929 16:58:29.558821 3501 sched.cpp:233] New master detected at master@192.168.122.164:55618 I0929 16:58:29.559054 3501 sched.cpp:283] Authenticating with master master@192.168.122.164:55618 I0929 16:58:29.559360 3501 authenticatee.hpp:128] Creating new client SASL connection I0929 16:58:29.560096 3501 master.cpp:3737] Authenticating scheduler-c8df3f3b-2552-476f-9daf-9aa2f012ad28@192.168.122.164:55618 I0929 16:58:29.560430 3501 authenticator.hpp:156] Creating new server SASL connection I0929 16:58:29.561141 3501 authenticatee.hpp:219] Received SASL authentication mechanisms: CRAM-MD5 I0929 16:58:29.561465 3501 authenticatee.hpp:245] Attempting to authenticate with mechanism 'CRAM-MD5' I0929 16:58:29.561743 3501 authenticator.hpp:262] Received SASL authentication start I0929 16:58:29.562098 3501 authenticator.hpp:384] Authentication requires more steps I0929 16:58:29.562353 3501 authenticatee.hpp:265] Received SASL authentication step I0929 16:58:29.562721 3507 authenticator.hpp:290] Received SASL authentication step I0929 16:58:29.563022 3507 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'fedora-20' server FQDN: 'fedora-20' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0929 16:58:29.563254 3507 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I0929 16:58:29.563484 3507 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0929 16:58:29.563736 3507 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'fedora-20' server FQDN: 'fedora-20' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0929 16:58:29.563976 3507 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0929 16:58:29.564188 3507 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0929 16:58:29.564415 3507 authenticator.hpp:376] Authentication success I0929 16:58:29.564673 3507 master.cpp:3777] Successfully authenticated principal 'test-principal' at scheduler-c8df3f3b-2552-476f-9daf-9aa2f012ad28@192.168.122.164:55618 I0929 16:58:29.568681 3501 authenticatee.hpp:305] Authentication success I0929 16:58:29.569046 3501 sched.cpp:357] Successfully authenticated with master master@192.168.122.164:55618 I0929 16:58:29.569286 3501 sched.cpp:476] Sending registration request to master@192.168.122.164:55618 I0929 16:58:29.569581 3507 master.cpp:1360] Received registration request from scheduler-c8df3f3b-2552-476f-9daf-9aa2f012ad28@192.168.122.164:55618 I0929 16:58:29.569846 3507 master.cpp:1320] Authorizing framework principal 'test-principal' to receive offers for role '*' I0929 16:58:29.570219 3507 master.cpp:1419] Registering framework 20140929-165829-2759502016-55618-3486-0000 at scheduler-c8df3f3b-2552-476f-9daf-9aa2f012ad28@192.168.122.164:55618 I0929 16:58:29.570543 3506 sched.cpp:407] Framework registered with 20140929-165829-2759502016-55618-3486-0000 I0929 16:58:29.570811 3506 sched.cpp:421] Scheduler::registered took 13811ns I0929 16:58:29.571135 3502 hierarchical_allocator_process.hpp:329] Added framework 20140929-165829-2759502016-55618-3486-0000 I0929 16:58:29.571393 3502 hierarchical_allocator_process.hpp:734] Offering cpus(*):2; mem(*):1024; disk(*):752; ports(*):[31000-32000] on slave 20140929-165829-2759502016-55618-3486-0 to framework 20140929-165829-2759502016-55618-3486-0000 I0929 16:58:29.571723 3502 hierarchical_allocator_process.hpp:659] Performed allocation for 1 slaves in 368547ns I0929 16:58:29.572125 3507 master.hpp:868] Adding offer 20140929-165829-2759502016-55618-3486-0 with resources cpus(*):2; mem(*):1024; disk(*):752; ports(*):[31000-32000] on slave 20140929-165829-2759502016-55618-3486-0 (fedora-20) I0929 16:58:29.572374 3507 master.cpp:3679] Sending 1 offers to framework 20140929-165829-2759502016-55618-3486-0000 I0929 16:58:29.572841 3503 sched.cpp:544] Scheduler::resourceOffers took 114306ns I0929 16:58:29.573197 3507 master.hpp:877] Removing offer 20140929-165829-2759502016-55618-3486-0 with resources cpus(*):2; mem(*):1024; disk(*):752; ports(*):[31000-32000] on slave 20140929-165829-2759502016-55618-3486-0 (fedora-20) I0929 16:58:29.573457 3507 master.cpp:2274] Processing reply for offers: [ 20140929-165829-2759502016-55618-3486-0 ] on slave 20140929-165829-2759502016-55618-3486-0 at slave(57)@192.168.122.164:55618 (fedora-20) for framework 20140929-165829-2759502016-55618-3486-0000 W0929 16:58:29.573717 3507 master.cpp:1944] Executor default for task 0 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases. W0929 16:58:29.573953 3507 master.cpp:1955] Executor default for task 0 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases. I0929 16:58:29.574177 3507 master.cpp:2357] Authorizing framework principal 'test-principal' to launch task 0 as user 'jenkins' I0929 16:58:29.574745 3507 master.hpp:845] Adding task 0 with resources cpus(*):2; mem(*):512 on slave 20140929-165829-2759502016-55618-3486-0 (fedora-20) I0929 16:58:29.574992 3507 master.cpp:2423] Launching task 0 of framework 20140929-165829-2759502016-55618-3486-0000 with resources cpus(*):2; mem(*):512 on slave 20140929-165829-2759502016-55618-3486-0 at slave(57)@192.168.122.164:55618 (fedora-20) I0929 16:58:29.575315 3503 slave.cpp:1023] Got assigned task 0 for framework 20140929-165829-2759502016-55618-3486-0000 I0929 16:58:29.575724 3503 slave.cpp:1133] Launching task 0 for framework 20140929-165829-2759502016-55618-3486-0000 I0929 16:58:29.578129 3503 exec.cpp:132] Version: 0.21.0 I0929 16:58:29.578505 3504 exec.cpp:182] Executor started at: executor(30)@192.168.122.164:55618 with pid 3486 I0929 16:58:29.578867 3503 slave.cpp:1246] Queuing task '0' for executor default of framework '20140929-165829-2759502016-55618-3486-0000 I0929 16:58:29.579144 3503 slave.cpp:554] Successfully attached file '/tmp/AllocatorTest_0_SlaveLost_xdXHfg/slaves/20140929-165829-2759502016-55618-3486-0/frameworks/20140929-165829-2759502016-55618-3486-0000/executors/default/runs/b0de9759-7054-4763-90f4-889ddc3a8524' I0929 16:58:29.579401 3503 slave.cpp:1756] Got registration for executor 'default' of framework 20140929-165829-2759502016-55618-3486-0000 fro... | 1 |
| 133 | MESOS-1853 | Remove /proc and /sys remounts from port_mapping isolator | /proc/net reflects a new network namespace regardless and remount doesn't actually do what we expected anyway, i.e., it's not sufficient for a new pid namespace and a new mount is required. | 3 |
| 134 | MESOS-1855 | Mesos 0.20.1 doesn't compile | The compilation of Mesos 0.20.1 fails on Ubuntu Trusty with the following error - slave/containerizer/mesos/containerizer.cpp -fPIC -DPIC -o slave/containerizer/mesos/.libs/libmesos_no_3rdparty_la-containerizer.o In file included from ./linux/routing/filter/ip.hpp:36:0, from ./slave/containerizer/isolators/network/port_mapping.hpp:42, from slave/containerizer/mesos/containerizer.cpp:44: ./linux/routing/filter/filter.hpp:29:43: fatal error: linux/routing/filter/handle.hpp: No such file or directory #include "linux/routing/filter/handle.hpp" ^ | 1 |
| 135 | MESOS-1856 | Support specifying libnl3 install location. | LIBNL_CFLAGS uses a hard-coded path in the configure script, instead of detecting the location. | 2 |
| 136 | MESOS-1858 | Leaked file descriptors in StatusUpdateStream. | https://github.com/apache/mesos/blob/master/src/slave/status_update_manager.hpp#L180 We should set cloexec for 'fd'. | 1 |
| 137 | MESOS-1862 | Performance regression in the Master's http metrics. | As part of the change to hold on to terminal unacknowledged tasks in the master, we introduced a performance regression during the following patch: https://github.com/apache/mesos/commit/0760b007ad65bc91e8cea377339978c78d36d247 {noformat} commit 0760b007ad65bc91e8cea377339978c78d36d247 Author: Benjamin Mahler <bmahler@twitter.com> Date: Thu Sep 11 10:48:20 2014 -0700 Minor cleanups to the Master code. Review: https://reviews.apache.org/r/25566 {noformat} Rather than keeping a running count of allocated resources, we now compute resources on-demand. This was done in order to ignore terminal task's resources. As a result of this change, the /stats.json and /metrics/snapshot endpoints on the master have slowed down substantially on large clusters. {noformat} $ time curl localhost:5050/health real 0m0.004s user 0m0.001s sys 0m0.002s $ time curl localhost:5050/stats.json > /dev/null real 0m15.402s user 0m0.001s sys 0m0.003s $ time curl localhost:5050/metrics/snapshot > /dev/null real 0m6.059s user 0m0.002s sys 0m0.002s {noformat} {{perf top}} reveals some of the resource computation during a request to stats.json: {noformat: perf top} Events: 36K cycles 10.53% libc-2.5.so [.] _int_free 9.90% libc-2.5.so [.] malloc 8.56% libmesos-0.21.0.so [.] std::_Rb_tree<process::ProcessBase*, process::ProcessBase*, std::_Identity<process::ProcessBase*>, std::less<process::ProcessBase*>, std::allocator<process::ProcessBase*> >:: 8.23% libc-2.5.so [.] _int_malloc 5.80% libstdc++.so.6.0.8 [.] std::_Rb_tree_increment(std::_Rb_tree_node_base*) 5.33% [kernel] [k] _raw_spin_lock 3.13% libstdc++.so.6.0.8 [.] std::string::assign(std::string const&) 2.95% libmesos-0.21.0.so [.] process::SocketManager::exited(process::ProcessBase*) 2.43% libmesos-0.21.0.so [.] mesos::Resource::MergeFrom(mesos::Resource const&) 1.88% libmesos-0.21.0.so [.] mesos::internal::master::Slave::used() const 1.48% libstdc++.so.6.0.8 [.] __gnu_cxx::__atomic_add(int volatile*, int) 1.45% [kernel] [k] find_busiest_group 1.41% libc-2.5.so [.] free 1.38% libmesos-0.21.0.so [.] mesos::Value_Range::MergeFrom(mesos::Value_Range const&) 1.13% libmesos-0.21.0.so [.] mesos::Value_Scalar::MergeFrom(mesos::Value_Scalar const&) 1.12% libmesos-0.21.0.so [.] mesos::Resource::SharedDtor() 1.07% libstdc++.so.6.0.8 [.] __gnu_cxx::__exchange_and_add(int volatile*, int) 0.94% libmesos-0.21.0.so [.] google::protobuf::UnknownFieldSet::MergeFrom(google::protobuf::UnknownFieldSet const&) 0.92% libstdc++.so.6.0.8 [.] operator new(unsigned long) 0.88% libmesos-0.21.0.so [.] mesos::Value_Ranges::MergeFrom(mesos::Value_Ranges const&) 0.75% libmesos-0.21.0.so [.] mesos::matches(mesos::Resource const&, mesos::Resource const&) {noformat} | 3 |
| 138 | MESOS-1863 | Split launch tasks and decline offers metrics | Both launchTasks() and declineOffers() scheduler driver calls end up in "messages_launch_tasks" metric on the master. It would be nice to split them for differentiating these two calls. | 1 |
| 139 | MESOS-1865 | Redirect to the leader master when current master is not a leader | Some of the API endpoints, for example /master/tasks.json, will return bogus information if you query a non-leading master: {code} [steven@Anesthetize:~]% curl http://master1.mesos-vpcqa.otenv.com:5050/master/tasks.json | jq . | head -n 10 { "tasks": [] } [steven@Anesthetize:~]% curl http://master2.mesos-vpcqa.otenv.com:5050/master/tasks.json | jq . | head -n 10 { "tasks": [] } [steven@Anesthetize:~]% curl http://master3.mesos-vpcqa.otenv.com:5050/master/tasks.json | jq . | head -n 10 { "tasks": [ { "executor_id": "", "framework_id": "20140724-231003-419644938-5050-1707-0000", "id": "pp.guestcenterwebhealthmonitor.606cd6ee-4b50-11e4-825b-5212e05f35db", "name": "pp.guestcenterwebhealthmonitor.606cd6ee-4b50-11e4-825b-5212e05f35db", "resources": { "cpus": 0.25, "disk": 0, {code} This is very hard for end-users to work around. For example if I query "which master is leading" followed by "leader: which tasks are running" it is possible that the leader fails over in between, leaving me with an incorrect answer and no way to know that this happened. In my opinion the API should return the correct response (by asking the current leader?) or an error (500 Not the leader?) but it's unacceptable to return a successful wrong answer. | 3 |
| 140 | MESOS-1866 | Race between ~Authenticator() and Authenticator::authenticate() can lead to schedulers/slaves to never get authenticated | The master might get a duplicate authenticate() request while a previous authentication attempt is in progress. Depending on what the AuthenticatorProcess is executing at the time, there are 2 possible race conditions which will cause scheduler/slave to continuously retry authentication but never succeed. We have seen both the race conditions in a heavily loaded production cluster. Race1: ---------- --> An authenticate() event was dispatched to AuthenticatorProcess (Master::authenticate() called Authenticator::authenticate()) --> A terminate() event was then injected into the front of the AuthenticatorProcess queue (duplicate Master::authenticate() did ~Authenticator) before the above authenticate() event was executed. --> Due to the bug in libprocess, the future returned by Master::authenticate() was never transitioned to discarded (Master::_authenticate() was never called). --> This caused all the subsequent authentication retries to be enqueued on the master waiting for Master::_authenticate() to be executed. Fix: Transition the dispatched future to discarded if the libprocess is terminated (https://reviews.apache.org/r/25945/) Race 2: ----------- --> An authenticate() event was dispatched to AuthenticatorProcess (Master::authenticate() called Authenticator::authenticate()) --> AuthenticatorProcess::authenticate() executed and set promise.onDiscard(defer(self, Self::discarded)). NOTE: The internal promise of AuthenticatorProcess is discarded in AuthenticatorProcess::discarded() --> A terminate() event was then injected into the front of the AuthenticatorProcess queue (duplicate Master::authenticate() did ~Authenticator) before the above discarded() event was executed) --> ~AuthenticatorProcess is destructed without ever discarding the internal promise (Master::_authenticate() was never called). --> This caused all the subsequent authentication retries to be enqueued on the master waiting for Master::_authenticate() to be executed. Fix: The fix here is to discard the internal promise when the AuthenticatorProcess is destructed. | 2 |
| 141 | MESOS-1869 | UpdateFramework message might reach the slave before Reregistered message and get dropped | In reregisterSlave() we send 'SlaveReregisteredMessage' before we link the slave pid, which means a temporary socket will be created and used. Subsequently, after linking, we send the UpdateFrameworkMessage, which creates and uses a persistent socket. This might lead to out-of-order delivery, resulting in UpdateFrameworkMessage reaching the slave before the SlaveReregisteredMessage and getting dropped because the slave is not yet (re-)registered. | 1 |
| 142 | MESOS-1875 | os::killtree() incorrectly returns early if pid has terminated | If groups == true and/or sessions == true then os::killtree() should continue to signal all processes in the process group and/or session, even if the leading pid has terminated. | 2 |
| 143 | MESOS-1901 | Slave resources obtained from localhost:5051/state.json is not correct. | The 'resources' field in Slave is uninitialized. Also, seems that 'attributes' field in Slave is redundant as we store slave info. | 2 |
| 144 | MESOS-1903 | Add backoff to framework re-registration retries | To avoid so many duplicate framework re-registration attempts (and thus offer rescinds) we should add backoff to re-registration retries. | 3 |
| 145 | MESOS-1913 | Create libevent/SSL-backed Socket implementation | NULL | 13 |
| 146 | MESOS-1941 | Make executor's user owner of executor's cgroup directory | Currently, when cgroups are enabled, and executor is spawned, it's mounted under, for ex: /sys/fs/cgroup/cpu/mesos/<mesos-id>. This directory in current implementation is only writable by root user. This prevents process launched by executor to mount its child processes under this cgroup, because the cgroup directory is only writable by root. To enable a executor spawned process to mount it's child processes under it's cgroup directory, the cgroup directory should be made writable by the user which spawns the executor. | 3 |
| 147 | MESOS-1943 | Add event queue size metrics to scheduler driver | In the master process, we expose metrics for event queue sizes for various event types. We should do the same for the scheduler driver process. | 2 |
| 148 | MESOS-1955 | Specification for Executor and Task life cycles in Slave | We should create a precise specification of what the Mesos source code is supposed to be implementing wrt. the life cycle of executors and tasks. And in addition, we should document why certain design decisions have been made one way or another, to provide guidance for future code changes. With such a source code-independent specification, we could write unbiased regression and scale tests, which would be instrumental in maintaining high quality. Furthermore, this would make the source code more amenable. Why pick this particular area of the source code? Shouldn't more of Mesos have a thorough specification? Probably so. But we need to start somewhere and this area seems to be a good choice, given both its intricacy and its importance. | 5 |
| 149 | MESOS-1964 | 0.21.0 release | Mesos release 0.21.0 will include the following major feature(s): - Provide state reconciliation for frameworks. [(MESOS-1407)|https://issues.apache.org/jira/browse/MESOS-1407] Possible features to include: - Isolation of system directories (/tmp) for Mesos containers [(MESOS-1586)|https://issues.apache.org/jira/browse/MESOS-1586] - Expose reason for TASK_KILLED [(1930)|https://issues.apache.org/jira/browse/MESOS-1930] This ticket will be used to track blockers to this release. | 5 |
| 150 | MESOS-1967 | Test RoutingTest.INETSockets fails on some machine | {noformat} [ RUN ] RoutingTest.INETSockets ../../../mesos/src/tests/routing_tests.cpp:238: Failure infos: Input data out of range ABORT: (../../../mesos/3rdparty/libprocess/3rdparty/stout/include/stout/try.hpp:92): Try::get() but state == ERROR: Input data out of range*** Aborted at 1414000937 (unix time) try "date -d @1414000937" if you are using GNU date *** PC: @ 0x7f2c2d509fc5 __GI_raise *** SIGABRT (@0x1b49000040b1) received by PID 16561 (TID 0x7f2c31031720) from PID 16561; stack trace: *** @ 0x7f2c2f0d4ca0 (unknown) @ 0x7f2c2d509fc5 __GI_raise @ 0x7f2c2d50ba70 __GI_abort @ 0x4cf782 _Abort() @ 0x4cf7bc _Abort() @ 0x99459e RoutingTest_INETSockets_Test::TestBody() @ 0xa1c363 testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0xa13617 testing::Test::Run() @ 0xa136be testing::TestInfo::Run() @ 0xa137c5 testing::TestCase::Run() @ 0xa13a68 testing::internal::UnitTestImpl::RunAllTests() @ 0xa13cf7 testing::UnitTest::Run() @ 0x49bc4b main @ 0x7f2c2d4f79f4 __libc_start_main @ 0x4aad79 (unknown) make[3]: *** [check-local] Aborted {noformat} | 2 |
| 151 | MESOS-1969 | RBT only takes revision ranges as args for versions >= 0.6 | the {{support/post-reviews.py}} script doesn't differentiate between RBT versions although the calling conventions for passing revision ranges are different. | 1 |
| 152 | MESOS-1970 | slave and offer ids are indistinguishable in the logs | It is currently impossible to tell slave ids and offer ids apart when looking at logs. Adding some differentiator will make log reading a little simpler. | 1 |
| 153 | MESOS-1972 | Move TASK_LOST generations due to invalid tasks from scheduler driver to master | As we move towards pure scheduler/executor clients, it is imperative that the scheduler driver doesn't do validation of tasks and generate TASK_LOST messages itself. All that logic should live in the master. Schedulers should reconcile dropped messages via reconciliation. | 3 |
| 154 | MESOS-1974 | Refactor the C++ Resources abstraction for DiskInfo | As we introduce DiskInfo and reservation for Resource. We need to change the C++ Resources abstraction to properly deal with merge/split of resources with those additional fields. Also, the existing C++ 'Resources' interfaces are poorly designed. Some of them are confusing and unintuitive. Some of them are overloaded with too many functionalities. For instance, {noformat} bool operator <= (const Resource& left, const Resource& right); {noformat} This interface in non-intuitive because A <= B doesn't imply !(B <= A). {noformat} Resource operator + (const Resource& left, const Resource& right); {noformat} This one is also non-intuitive because if 'left' is not compatible with 'right', the result is 'left' (why not right???). Similar for operator '-'. {noformat} Option<Resource> Resources::get(const Resource& r) const; {noformat} This one assume Resources is flattened, but it might not be. As we start to introduce persistent disk resources (MESOS-1554), things will get more complicated. For example, one may want to get two types of 'disk()' functions: one returns the ephemeral disk bytes (with no disk info), one returns the total disk bytes (including ones that have disk info). We may wanna introduce a concept about Resource that indicates that a resource cannot be merged or split (e.g., atomic?). Since we need to change this class anyway. I wanna take this chance to refactor it. | 8 |
| 155 | MESOS-1984 | Documentation for Egress Control Limit | NULL | 1 |
| 156 | MESOS-1989 | Container network stats reported by the port mapping isolator is the reverse of the actual network stats. | Looks like the TX/RX network stats reported is the reverse of the actual network stats. The reason is because we simply get TX/RX data from veth on the host. Since veth pair is a tunnel, the ingress of veth on host is the egress of eth0 in container (and vice versa). Therefore, we need to flip the data we got from veth. {noformat} [jyu@... ~]$ sudo ip netns exec 24926 /sbin/ip -s link show dev eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000 link/ether f0:4d:a2:75:74:05 brd ff:ff:ff:ff:ff:ff RX: bytes packets errors dropped overrun mcast 46030857691178 12561038581 0 0 0 0 TX: bytes packets errors dropped carrier collsns 29792886058561 15036798198 0 0 0 0 [jyu@... ~]$ ip -s link show dev mesos24926 7412: mesos24926: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000 link/ether f0:4d:a2:75:74:05 brd ff:ff:ff:ff:ff:ff RX: bytes packets errors dropped overrun mcast 29793066979551 15036894749 0 0 0 0 TX: bytes packets errors dropped carrier collsns 46031126366116 12561113732 0 0 0 0 {noformat} | 1 |
| 157 | MESOS-2007 | AllocatorTest/0.SlaveReregistersFirst is flaky | {noformat:title=} [ RUN ] AllocatorTest/0.SlaveReregistersFirst Using temporary directory '/tmp/AllocatorTest_0_SlaveReregistersFirst_YPe61d' I1028 23:48:22.360447 31190 leveldb.cpp:176] Opened db in 2.192575ms I1028 23:48:22.361253 31190 leveldb.cpp:183] Compacted db in 760753ns I1028 23:48:22.361320 31190 leveldb.cpp:198] Created db iterator in 22188ns I1028 23:48:22.361340 31190 leveldb.cpp:204] Seeked to beginning of db in 1950ns I1028 23:48:22.361351 31190 leveldb.cpp:273] Iterated through 0 keys in the db in 345ns I1028 23:48:22.361403 31190 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I1028 23:48:22.362185 31217 recover.cpp:437] Starting replica recovery I1028 23:48:22.362764 31219 recover.cpp:463] Replica is in EMPTY status I1028 23:48:22.363955 31210 replica.cpp:638] Replica in EMPTY status received a broadcasted recover request I1028 23:48:22.364320 31217 recover.cpp:188] Received a recover response from a replica in EMPTY status I1028 23:48:22.364820 31211 recover.cpp:554] Updating replica status to STARTING I1028 23:48:22.365365 31215 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 418991ns I1028 23:48:22.365391 31215 replica.cpp:320] Persisted replica status to STARTING I1028 23:48:22.365617 31217 recover.cpp:463] Replica is in STARTING status I1028 23:48:22.366328 31206 master.cpp:312] Master 20141028-234822-3193029443-50043-31190 (pietas.apache.org) started on 67.195.81.190:50043 I1028 23:48:22.366377 31206 master.cpp:358] Master only allowing authenticated frameworks to register I1028 23:48:22.366391 31206 master.cpp:363] Master only allowing authenticated slaves to register I1028 23:48:22.366402 31206 credentials.hpp:36] Loading credentials for authentication from '/tmp/AllocatorTest_0_SlaveReregistersFirst_YPe61d/credentials' I1028 23:48:22.366708 31206 master.cpp:392] Authorization enabled I1028 23:48:22.366886 31209 replica.cpp:638] Replica in STARTING status received a broadcasted recover request I1028 23:48:22.367311 31208 master.cpp:120] No whitelist given. Advertising offers for all slaves I1028 23:48:22.367312 31207 recover.cpp:188] Received a recover response from a replica in STARTING status I1028 23:48:22.367686 31211 hierarchical_allocator_process.hpp:299] Initializing hierarchical allocator process with master : master@67.195.81.190:50043 I1028 23:48:22.367863 31212 recover.cpp:554] Updating replica status to VOTING I1028 23:48:22.368477 31218 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 375527ns I1028 23:48:22.368505 31218 replica.cpp:320] Persisted replica status to VOTING I1028 23:48:22.368517 31204 master.cpp:1242] The newly elected leader is master@67.195.81.190:50043 with id 20141028-234822-3193029443-50043-31190 I1028 23:48:22.368549 31204 master.cpp:1255] Elected as the leading master! I1028 23:48:22.368567 31204 master.cpp:1073] Recovering from registrar I1028 23:48:22.368621 31215 recover.cpp:568] Successfully joined the Paxos group I1028 23:48:22.368716 31219 registrar.cpp:313] Recovering registrar I1028 23:48:22.369000 31215 recover.cpp:452] Recover process terminated I1028 23:48:22.369523 31208 log.cpp:656] Attempting to start the writer I1028 23:48:22.370909 31205 replica.cpp:474] Replica received implicit promise request with proposal 1 I1028 23:48:22.371266 31205 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 325016ns I1028 23:48:22.371290 31205 replica.cpp:342] Persisted promised to 1 I1028 23:48:22.371979 31218 coordinator.cpp:230] Coordinator attemping to fill missing position I1028 23:48:22.373378 31210 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I1028 23:48:22.373746 31210 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 329018ns I1028 23:48:22.373772 31210 replica.cpp:676] Persisted action at 0 I1028 23:48:22.374897 31214 replica.cpp:508] Replica received write request for position 0 I1028 23:48:22.374951 31214 leveldb.cpp:438] Reading position from leveldb took 26002ns I1028 23:48:22.375272 31214 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 289094ns I1028 23:48:22.375298 31214 replica.cpp:676] Persisted action at 0 I1028 23:48:22.375886 31204 replica.cpp:655] Replica received learned notice for position 0 I1028 23:48:22.376258 31204 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 346650ns I1028 23:48:22.376277 31204 replica.cpp:676] Persisted action at 0 I1028 23:48:22.376298 31204 replica.cpp:661] Replica learned NOP action at position 0 I1028 23:48:22.376843 31215 log.cpp:672] Writer started with ending position 0 I1028 23:48:22.378056 31205 leveldb.cpp:438] Reading position from leveldb took 28265ns I1028 23:48:22.380323 31217 registrar.cpp:346] Successfully fetched the registry (0B) in 11.55584ms I1028 23:48:22.380466 31217 registrar.cpp:445] Applied 1 operations in 50632ns; attempting to update the 'registry' I1028 23:48:22.382472 31217 log.cpp:680] Attempting to append 139 bytes to the log I1028 23:48:22.382715 31210 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I1028 23:48:22.383463 31210 replica.cpp:508] Replica received write request for position 1 I1028 23:48:22.383857 31210 leveldb.cpp:343] Persisting action (158 bytes) to leveldb took 363758ns I1028 23:48:22.383875 31210 replica.cpp:676] Persisted action at 1 I1028 23:48:22.384397 31218 replica.cpp:655] Replica received learned notice for position 1 I1028 23:48:22.384840 31218 leveldb.cpp:343] Persisting action (160 bytes) to leveldb took 420161ns I1028 23:48:22.384862 31218 replica.cpp:676] Persisted action at 1 I1028 23:48:22.384882 31218 replica.cpp:661] Replica learned APPEND action at position 1 I1028 23:48:22.385684 31211 registrar.cpp:490] Successfully updated the 'registry' in 5.158144ms I1028 23:48:22.385818 31211 registrar.cpp:376] Successfully recovered registrar I1028 23:48:22.385912 31214 log.cpp:699] Attempting to truncate the log to 1 I1028 23:48:22.386101 31218 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I1028 23:48:22.386124 31211 master.cpp:1100] Recovered 0 slaves from the Registry (101B) ; allowing 10mins for slaves to re-register I1028 23:48:22.387398 31209 replica.cpp:508] Replica received write request for position 2 I1028 23:48:22.387758 31209 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 334969ns I1028 23:48:22.387776 31209 replica.cpp:676] Persisted action at 2 I1028 23:48:22.388272 31204 replica.cpp:655] Replica received learned notice for position 2 I1028 23:48:22.388453 31204 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 159390ns I1028 23:48:22.388501 31204 leveldb.cpp:401] Deleting ~1 keys from leveldb took 30409ns I1028 23:48:22.388516 31204 replica.cpp:676] Persisted action at 2 I1028 23:48:22.388531 31204 replica.cpp:661] Replica learned TRUNCATE action at position 2 I1028 23:48:22.400737 31207 slave.cpp:169] Slave started on 34)@67.195.81.190:50043 I1028 23:48:22.400786 31207 credentials.hpp:84] Loading credential for authentication from '/tmp/AllocatorTest_0_SlaveReregistersFirst_QPPV21/credential' I1028 23:48:22.400996 31207 slave.cpp:276] Slave using credential for: test-principal I1028 23:48:22.401304 31207 slave.cpp:289] Slave resources: cpus(*):2; mem(*):1024; disk(*):3.70122e+06; ports(*):[31000-32000] I1028 23:48:22.401413 31207 slave.cpp:318] Slave hostname: pietas.apache.org I1028 23:48:22.401520 31207 slave.cpp:319] Slave checkpoint: false W1028 23:48:22.401535 31207 slave.cpp:321] Disabling checkpointing is deprecated and the --checkpoint flag will be removed in a future release. Please avoid using this flag I1028 23:48:22.402349 31207 state.cpp:33] Recovering state from '/tmp/AllocatorTest_0_SlaveReregistersFirst_QPPV21/meta' I1028 23:48:22.402678 31207 status_update_manager.cpp:197] Recovering status update manager I1028 23:48:22.403048 31211 slave.cpp:3456] Finished recovery I1028 23:48:22.403815 31215 slave.cpp:602] New master detected at master@67.195.81.190:50043 I1028 23:48:22.403852 31215 slave.cpp:665] Authenticating with master master@67.195.81.190:50043 I1028 23:48:22.403875 31206 status_update_manager.cpp:171] Pausing sending status updates I1028 23:48:22.403961 31215 slave.cpp:638] Detecting new master I1028 23:48:22.404016 31211 authenticatee.hpp:133] Creating new client SASL connection I1028 23:48:22.404230 31204 master.cpp:3853] Authenticating slave(34)@67.195.81.190:50043 I1028 23:48:22.404464 31205 authenticator.hpp:161] Creating new server SASL connection I1028 23:48:22.404613 31211 authenticatee.hpp:224] Received SASL authentication mechanisms: CRAM-MD5 I1028 23:48:22.404649 31211 authenticatee.hpp:250] Attempting to authenticate with mechanism 'CRAM-MD5' I1028 23:48:22.404734 31211 authenticator.hpp:267] Received SASL authentication start I1028 23:48:22.404783 31211 authenticator.hpp:389] Authentication requires more steps I1028 23:48:22.404898 31215 authenticatee.hpp:270] Received SASL authentication step I1028 23:48:22.404999 31215 authenticator.hpp:295] Received SASL authentication step I1028 23:48:22.405030 31215 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pietas.apache.org' server FQDN: 'pietas.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I1028 23:48:22.405047 31215 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I1028 23:48:22.405086 31215 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I1028 23:48:22.405109 31215 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pietas.apache.org' server FQDN: 'pietas.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I1028 23:48:22.405122 31215 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I1028 23:48:22.405129 31215 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I1028 23:48:22.405146 31215 authenticator.hpp:381] Authentication success I1028 23:48:22.405243 31213 authenticatee.hpp:310] Authentication success I1028 23:48:22.405253 31214 master.cpp:3893] Successfully authenticated principal 'test-principal' at slave(34)@67.195.81.190:50043 I1028 23:48:22.405505 31213 slave.cpp:722] Successfully authenticated with master master@67.195.81.190:50043 I1028 23:48:22.405619 31213 slave.cpp:1050] Will retry registration in 17.050994ms if necessary I1028 23:48:22.405819 31215 master.cpp:3032] Registering slave at slave(34)@67.195.81.190:50043 (pietas.apache.org) with id 20141028-234822-3193029443-50043-31190-S0 I1028 23:48:22.406262 31216 registrar.cpp:445] Applied 1 operations in 52647ns; attempting to update the 'registry' I1028 23:48:22.406697 31190 sched.cpp:137] Version: 0.21.0 I1028 23:48:22.407083 31211 sched.cpp:233] New master detected at master@67.195.81.190:50043 I1028 23:48:22.407114 31211 sched.cpp:283] Authenticating with master master@67.195.81.190:50043 I1028 23:48:22.407290 31214 authenticatee.hpp:133] Creating new client SASL connection I1028 23:48:22.407424 31214 master.cpp:3853] Authenticating scheduler-0aa33fc7-0d29-487c-80eb-f933681f9c95@67.195.81.190:50043 I1028 23:48:22.407659 31207 authenticator.hpp:161] Creating new server SASL connection I1028 23:48:22.407757 31207 authenticatee.hpp:224] Received SASL authentication mechanisms: CRAM-MD5 I1028 23:48:22.407774 31207 authenticatee.hpp:250] Attempting to authenticate with mechanism 'CRAM-MD5' I1028 23:48:22.407830 31207 authenticator.hpp:267] Received SASL authentication start I1028 23:48:22.407868 31207 authenticator.hpp:389] Authentication requires more steps I1028 23:48:22.407927 31207 authenticatee.hpp:270] Received SASL authentication step I1028 23:48:22.408015 31212 authenticator.hpp:295] Received SASL authentication step I1028 23:48:22.408037 31212 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pietas.apache.org' server FQDN: 'pietas.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I1028 23:48:22.408046 31212 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I1028 23:48:22.408072 31212 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I1028 23:48:22.408092 31212 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pietas.apache.org' server FQDN: 'pietas.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I1028 23:48:22.408100 31212 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I1028 23:48:22.408105 31212 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I1028 23:48:22.408116 31212 authenticator.hpp:381] Authentication success I1028 23:48:22.408192 31210 authenticatee.hpp:310] Authentication success I1028 23:48:22.408210 31217 master.cpp:3893] Successfully authenticated principal 'test-principal' at scheduler-0aa33fc7-0d29-487c-80eb-f933681f9c95@67.195.81.190:50043 I1028 23:48:22.408419 31210 sched.cpp:357] Successfully authenticated with master master@67.195.81.190:50043 I1028 23:48:22.408460 31210 sched.cpp:476] Sending registration request to master@67.195.81.190:50043 I1028 23:48:22.408568 31217 master.cpp:1362] Received registration request for framework 'default' at scheduler-0aa33fc7-0d29-487c-80eb-f933681f9c95@67.195.81.190:50043 I1028 23:48:22.408617 31217 master.cpp:1321] Authorizing framework principal 'test-principal' to receive offers for role '*' I1028 23:48:22.408937 31214 master.cpp:1426] Registering framework 20141028-234822-3193029443-50043-31190-0000 (default) at scheduler-0aa33fc7-0d29-487c-80eb-f933681f9c95@67.195.81.190:50043 I1028 23:48:22.409265 31213 sched.cpp:407] Framework registered with 20141028-234822-3193029443-50043-31190-0000 I1028 23:48:22.409267 31212 hierarchical_allocator_process.hpp:329] Added framework 20141028-234822-3193029443-50043-31190-0000 I1028 23:48:22.409312 31212 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1028 23:48:22.409324 31215 log.cpp:680] Attempting to append 316 bytes to the log I1028 23:48:22.409333 31213 sched.cpp:421] Scheduler::registered took 38591ns I1028 23:48:22.409327 31212 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 24107ns I1028 23:48:22.409518 31205 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I1028 23:48:22.410127 31206 replica.cpp:508] Replica received write request for position 3 I1028 23:48:22.410706 31206 leveldb.cpp:343] Persisting action (335 bytes) to leveldb took 554098ns I1028 23:48:22.410725 31206 replica.cpp:676] Persisted action at 3 I1028 23:48:22.411151 31217 replica.cpp:655] Replica received learned notice for position 3 I1028 23:48:22.411499 31217 leveldb.cpp:343] Persisting action (337 bytes) to leveldb took 326572ns I1028 23:48:22.411519 31217 replica.cpp:676] Persisted action at 3 I1028 23:48:22.411533 31217 replica.cpp:661] Replica learned APPEND action at position 3 I1028 23:48:22.412292 31219 registrar.cpp:490] Successfully updated the 'registry' in 5.972992ms I1028 23:48:22.412518 31218 log.cpp:699] Attempting to truncate the log to 3 I1028 23:48:22.412621 31213 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 I1028 23:48:22.412734 31219 slave.cpp:2522] Received ping from slave-observer(38)@67.195.81.190:50043 I1028 23:48:22.412787 31206 master.cpp:3086] Registered slave 20141028-234822-3193029443-50043-31190-S0 at slave(34)@67.195.81.190:50043 (pietas.apache.org) with cpus(*):2; mem(*):1024; disk(*):3.70122e+06; ports(*):[31000-32000] I1028 23:48:22.412858 31219 slave.cpp:756] Registered with master master@67.195.81.190:50043; given slave ID 20141028-234822-3193029443-50043-31190-S0 I1028 23:48:22.412994 31210 status_update_manager.cpp:178] Resuming sending status updates I1028 23:48:22.413014 31211 hierarchical_allocator_process.hpp:442] Added slave 20141028-234822-3193029443-50043-31190-S0 (pietas.apache.org) with cpus(*):2; mem(*):1024; disk(*):3.70122e+06; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):3.70122e+06; ports(*):[31000-32000] available) I1028 23:48:22.413159 31211 hierarchical_allocator_process.hpp:734] Offering cpus(*):2; mem(*):1024; disk(*):3.70122e+06; ports(*):[31000-32000] on slave 20141028-234822-3193029443-50043-31190-S0 to framework 20141028-234822-3193029443-50043-31190-0000 I1028 23:48:22.413290 31208 replica.cpp:508] Replica received write request for position 4 I1028 23:48:22.413421 31211 hierarchical_allocator_process.hpp:679] Performed allocation for slave 20141028-234822-3193029443-50043-31190-S0 in 346658ns I1028 23:48:22.413650 31208 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 336067ns I1028 23:48:22.413668 31208 replica.cpp:676] Persisted action at 4 I1028 23:48:22.413797 31216 master.cpp:3795] Sending 1 offers to framework 20141028-234822-3193029443-50043-31190-0000 (default) at scheduler-0aa33fc7-0d29-487c-80eb-f933681f9c95@67.195.81.190:50043 I1028 23:48:22.414077 31212 replica.cpp:655] Replica received learned notice for position 4 I1028 23:48:22.414356 31212 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 260401ns I1028 23:48:22.414403 31212 leveldb.cpp:401] Deleting ~2 keys from leveldb took 28541ns I1028 23:48:22.414417 31212 replica.cpp:676] Persisted action at 4 I1028 23:48:22.414446 31212 replica.cpp:661] Replica learned TRUNCATE action at position 4 I1028 23:48:22.414422 31207 sched.cpp:544] Scheduler::resourceOffers took 310278ns I1028 23:48:22.415086 31214 master.cpp:2321] Processing reply for offers: [ 20141028-234822-3193029443-50043-31190-O0 ] on slave 20141028-234822-3193029443-50043-31190-S0 at slave(34)@67.195.81.190:50043 (pietas.apache.org) for framework 20141028-234822-3193029443-50043-31190-0000 (default) at scheduler-0aa33fc7-0d29-487c-80eb-f933681f9c95@67.195.81.190:50043 W1028 23:48:22.415163 31214 master.cpp:1969] Executor default for task 0 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases. W1028 23:48:22.415186 31214 master.cpp:1980] Executor default for task 0 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases. I1028 23:48:22.415256 31214 master.cpp:2417] Authorizing framework principal 'test-principal' to launch task 0 as user 'jenkins' I1028 23:48:22.416033 31219 master.hpp:877] Adding task 0 with resources cpus(*):1; mem(*):500 on slave 20141028-234822-3193029443-50043-31190-S0 (pietas.apache.org) I1028 23:48:22.416084 31219 master.cpp:2480] Launching task 0 of framework 20141028-234822-3193029443-50043-31190-0000 (default) at scheduler-0aa33fc7-0d29-487c-80eb-f933681f9c95@67.195.81.190:50043 with resources cpus(*):1; mem(*):500 on slave 20141028-234822-3193029443-50043-31190-S0 at slave(34)@67.195.81.190:50043 (pietas.apache.org) I1028 23:48:22.416317 31214 slave.cpp:1081] Got assigned task 0 for framework 20141028-234822-3193029443-50043-31190-0000 I1028 23:48:22.416679 31215 hierarchical_allocator_process.hpp:563] Recovered cpus(*):1; mem(*):524; disk(*):3.70122e+06; ports(*):[31000-32000] (total allocatable: cpus(*):1; mem(*):524; disk(*):3.70122e+06; ports(*):[31000-32000]) on slave 20141028-234822-3193029443-50043-31190-S0 from framework 20141028-234822-3193029443-50043-31190-0000 I1028 23:48:22.416721 31215 hierarchical_allocator_process.hpp:599] Framework 20141028-234822-3193029443-50043-31190-0000 filtered slave 2... | 2 |
| 158 | MESOS-2008 | MasterAuthorizationTest.DuplicateReregistration is flaky | {noformat:title=} [ RUN ] MasterAuthorizationTest.DuplicateReregistration Using temporary directory '/tmp/MasterAuthorizationTest_DuplicateReregistration_DLOmYX' I1029 08:25:26.021766 32232 leveldb.cpp:176] Opened db in 3.066621ms I1029 08:25:26.022734 32232 leveldb.cpp:183] Compacted db in 935019ns I1029 08:25:26.022766 32232 leveldb.cpp:198] Created db iterator in 4350ns I1029 08:25:26.022785 32232 leveldb.cpp:204] Seeked to beginning of db in 902ns I1029 08:25:26.022799 32232 leveldb.cpp:273] Iterated through 0 keys in the db in 387ns I1029 08:25:26.022831 32232 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I1029 08:25:26.023305 32248 recover.cpp:437] Starting replica recovery I1029 08:25:26.023598 32248 recover.cpp:463] Replica is in EMPTY status I1029 08:25:26.025059 32260 replica.cpp:638] Replica in EMPTY status received a broadcasted recover request I1029 08:25:26.025320 32247 recover.cpp:188] Received a recover response from a replica in EMPTY status I1029 08:25:26.025585 32256 recover.cpp:554] Updating replica status to STARTING I1029 08:25:26.026546 32249 master.cpp:312] Master 20141029-082526-3142697795-40696-32232 (pomona.apache.org) started on 67.195.81.187:40696 I1029 08:25:26.026561 32261 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 694444ns I1029 08:25:26.026592 32249 master.cpp:358] Master only allowing authenticated frameworks to register I1029 08:25:26.026592 32261 replica.cpp:320] Persisted replica status to STARTING I1029 08:25:26.026605 32249 master.cpp:363] Master only allowing authenticated slaves to register I1029 08:25:26.026639 32249 credentials.hpp:36] Loading credentials for authentication from '/tmp/MasterAuthorizationTest_DuplicateReregistration_DLOmYX/credentials' I1029 08:25:26.026877 32249 master.cpp:392] Authorization enabled I1029 08:25:26.026901 32260 recover.cpp:463] Replica is in STARTING status I1029 08:25:26.027498 32261 master.cpp:120] No whitelist given. Advertising offers for all slaves I1029 08:25:26.027541 32248 hierarchical_allocator_process.hpp:299] Initializing hierarchical allocator process with master : master@67.195.81.187:40696 I1029 08:25:26.028055 32252 replica.cpp:638] Replica in STARTING status received a broadcasted recover request I1029 08:25:26.028451 32247 recover.cpp:188] Received a recover response from a replica in STARTING status I1029 08:25:26.028733 32249 master.cpp:1242] The newly elected leader is master@67.195.81.187:40696 with id 20141029-082526-3142697795-40696-32232 I1029 08:25:26.028764 32249 master.cpp:1255] Elected as the leading master! I1029 08:25:26.028781 32249 master.cpp:1073] Recovering from registrar I1029 08:25:26.028904 32246 recover.cpp:554] Updating replica status to VOTING I1029 08:25:26.029163 32257 registrar.cpp:313] Recovering registrar I1029 08:25:26.029556 32251 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 485711ns I1029 08:25:26.029588 32251 replica.cpp:320] Persisted replica status to VOTING I1029 08:25:26.029726 32253 recover.cpp:568] Successfully joined the Paxos group I1029 08:25:26.029932 32253 recover.cpp:452] Recover process terminated I1029 08:25:26.030436 32250 log.cpp:656] Attempting to start the writer I1029 08:25:26.032152 32248 replica.cpp:474] Replica received implicit promise request with proposal 1 I1029 08:25:26.032778 32248 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 597030ns I1029 08:25:26.032807 32248 replica.cpp:342] Persisted promised to 1 I1029 08:25:26.033481 32254 coordinator.cpp:230] Coordinator attemping to fill missing position I1029 08:25:26.035429 32247 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I1029 08:25:26.036154 32247 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 690208ns I1029 08:25:26.036181 32247 replica.cpp:676] Persisted action at 0 I1029 08:25:26.037344 32249 replica.cpp:508] Replica received write request for position 0 I1029 08:25:26.037395 32249 leveldb.cpp:438] Reading position from leveldb took 22607ns I1029 08:25:26.038074 32249 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 647429ns I1029 08:25:26.038105 32249 replica.cpp:676] Persisted action at 0 I1029 08:25:26.038683 32247 replica.cpp:655] Replica received learned notice for position 0 I1029 08:25:26.039378 32247 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 664911ns I1029 08:25:26.039407 32247 replica.cpp:676] Persisted action at 0 I1029 08:25:26.039433 32247 replica.cpp:661] Replica learned NOP action at position 0 I1029 08:25:26.040045 32252 log.cpp:672] Writer started with ending position 0 I1029 08:25:26.041378 32251 leveldb.cpp:438] Reading position from leveldb took 25625ns I1029 08:25:26.044642 32246 registrar.cpp:346] Successfully fetched the registry (0B) in 15.433984ms I1029 08:25:26.044742 32246 registrar.cpp:445] Applied 1 operations in 16444ns; attempting to update the 'registry' I1029 08:25:26.047538 32256 log.cpp:680] Attempting to append 139 bytes to the log I1029 08:25:26.156330 32247 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I1029 08:25:26.158460 32261 replica.cpp:508] Replica received write request for position 1 I1029 08:25:26.159277 32261 leveldb.cpp:343] Persisting action (158 bytes) to leveldb took 782308ns I1029 08:25:26.159328 32261 replica.cpp:676] Persisted action at 1 I1029 08:25:26.160267 32255 replica.cpp:655] Replica received learned notice for position 1 I1029 08:25:26.161070 32255 leveldb.cpp:343] Persisting action (160 bytes) to leveldb took 750259ns I1029 08:25:26.161100 32255 replica.cpp:676] Persisted action at 1 I1029 08:25:26.161125 32255 replica.cpp:661] Replica learned APPEND action at position 1 I1029 08:25:26.162199 32253 registrar.cpp:490] Successfully updated the 'registry' in 117.40416ms I1029 08:25:26.162400 32253 registrar.cpp:376] Successfully recovered registrar I1029 08:25:26.162724 32249 master.cpp:1100] Recovered 0 slaves from the Registry (101B) ; allowing 10mins for slaves to re-register I1029 08:25:26.162757 32253 log.cpp:699] Attempting to truncate the log to 1 I1029 08:25:26.162919 32256 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I1029 08:25:26.163949 32250 replica.cpp:508] Replica received write request for position 2 I1029 08:25:26.164589 32250 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 603175ns I1029 08:25:26.164618 32250 replica.cpp:676] Persisted action at 2 I1029 08:25:26.165385 32251 replica.cpp:655] Replica received learned notice for position 2 I1029 08:25:26.166007 32251 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 594003ns I1029 08:25:26.166056 32251 leveldb.cpp:401] Deleting ~1 keys from leveldb took 23309ns I1029 08:25:26.166077 32251 replica.cpp:676] Persisted action at 2 I1029 08:25:26.166100 32251 replica.cpp:661] Replica learned TRUNCATE action at position 2 I1029 08:25:26.178493 32232 sched.cpp:137] Version: 0.21.0 I1029 08:25:26.179029 32256 sched.cpp:233] New master detected at master@67.195.81.187:40696 I1029 08:25:26.179078 32256 sched.cpp:283] Authenticating with master master@67.195.81.187:40696 I1029 08:25:26.179424 32246 authenticatee.hpp:133] Creating new client SASL connection I1029 08:25:26.179678 32259 master.cpp:3853] Authenticating scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:26.179970 32250 authenticator.hpp:161] Creating new server SASL connection I1029 08:25:26.180165 32250 authenticatee.hpp:224] Received SASL authentication mechanisms: CRAM-MD5 I1029 08:25:26.180191 32250 authenticatee.hpp:250] Attempting to authenticate with mechanism 'CRAM-MD5' I1029 08:25:26.180272 32250 authenticator.hpp:267] Received SASL authentication start I1029 08:25:26.180378 32250 authenticator.hpp:389] Authentication requires more steps I1029 08:25:26.180557 32260 authenticatee.hpp:270] Received SASL authentication step I1029 08:25:26.180704 32254 authenticator.hpp:295] Received SASL authentication step I1029 08:25:26.180737 32254 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I1029 08:25:26.180748 32254 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I1029 08:25:26.180780 32254 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I1029 08:25:26.180804 32254 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I1029 08:25:26.180816 32254 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I1029 08:25:26.180824 32254 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I1029 08:25:26.180841 32254 authenticator.hpp:381] Authentication success I1029 08:25:26.180937 32259 authenticatee.hpp:310] Authentication success I1029 08:25:26.180991 32260 master.cpp:3893] Successfully authenticated principal 'test-principal' at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:26.181422 32259 sched.cpp:357] Successfully authenticated with master master@67.195.81.187:40696 I1029 08:25:26.181449 32259 sched.cpp:476] Sending registration request to master@67.195.81.187:40696 I1029 08:25:26.181697 32260 master.cpp:1362] Received registration request for framework 'default' at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:26.181758 32260 master.cpp:1321] Authorizing framework principal 'test-principal' to receive offers for role '*' I1029 08:25:26.182063 32260 master.cpp:1426] Registering framework 20141029-082526-3142697795-40696-32232-0000 (default) at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:26.182430 32248 hierarchical_allocator_process.hpp:329] Added framework 20141029-082526-3142697795-40696-32232-0000 I1029 08:25:26.182462 32248 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:26.182462 32261 sched.cpp:407] Framework registered with 20141029-082526-3142697795-40696-32232-0000 I1029 08:25:26.182473 32248 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 15372ns I1029 08:25:26.182554 32261 sched.cpp:421] Scheduler::registered took 60059ns I1029 08:25:26.185515 32260 sched.cpp:227] Scheduler::disconnected took 16607ns I1029 08:25:26.185538 32260 sched.cpp:233] New master detected at master@67.195.81.187:40696 I1029 08:25:26.185567 32260 sched.cpp:283] Authenticating with master master@67.195.81.187:40696 I1029 08:25:26.185783 32246 authenticatee.hpp:133] Creating new client SASL connection I1029 08:25:26.186218 32250 master.cpp:3853] Authenticating scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:26.186456 32247 authenticator.hpp:161] Creating new server SASL connection I1029 08:25:26.186594 32250 authenticatee.hpp:224] Received SASL authentication mechanisms: CRAM-MD5 I1029 08:25:26.186621 32250 authenticatee.hpp:250] Attempting to authenticate with mechanism 'CRAM-MD5' I1029 08:25:26.186745 32259 authenticator.hpp:267] Received SASL authentication start I1029 08:25:26.186800 32259 authenticator.hpp:389] Authentication requires more steps I1029 08:25:26.186936 32260 authenticatee.hpp:270] Received SASL authentication step I1029 08:25:26.187062 32249 authenticator.hpp:295] Received SASL authentication step I1029 08:25:26.187095 32249 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I1029 08:25:26.187108 32249 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I1029 08:25:26.187137 32249 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I1029 08:25:26.187162 32249 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I1029 08:25:26.187175 32249 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I1029 08:25:26.187182 32249 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I1029 08:25:26.187199 32249 authenticator.hpp:381] Authentication success I1029 08:25:26.187327 32249 authenticatee.hpp:310] Authentication success I1029 08:25:26.187366 32260 master.cpp:3893] Successfully authenticated principal 'test-principal' at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:26.187631 32249 sched.cpp:357] Successfully authenticated with master master@67.195.81.187:40696 I1029 08:25:26.187659 32249 sched.cpp:476] Sending registration request to master@67.195.81.187:40696 I1029 08:25:27.028445 32251 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:28.045682 32251 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 1.017231941secs I1029 08:25:28.045760 32249 sched.cpp:476] Sending registration request to master@67.195.81.187:40696 I1029 08:25:28.045900 32253 master.cpp:1499] Received re-registration request from framework 20141029-082526-3142697795-40696-32232-0000 (default) at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:28.045989 32253 master.cpp:1321] Authorizing framework principal 'test-principal' to receive offers for role '*' I1029 08:25:28.046455 32253 master.cpp:1499] Received re-registration request from framework 20141029-082526-3142697795-40696-32232-0000 (default) at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:28.046529 32253 master.cpp:1321] Authorizing framework principal 'test-principal' to receive offers for role '*' I1029 08:25:28.050155 32247 sched.cpp:233] New master detected at master@67.195.81.187:40696 I1029 08:25:28.050217 32247 sched.cpp:283] Authenticating with master master@67.195.81.187:40696 I1029 08:25:28.050405 32252 master.cpp:1552] Re-registering framework 20141029-082526-3142697795-40696-32232-0000 (default) at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:28.050509 32253 authenticatee.hpp:133] Creating new client SASL connection I1029 08:25:28.050566 32252 master.cpp:1592] Allowing framework 20141029-082526-3142697795-40696-32232-0000 (default) at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 to re-register with an already used id I1029 08:25:28.051084 32257 sched.cpp:449] Framework re-registered with 20141029-082526-3142697795-40696-32232-0000 I1029 08:25:28.051151 32252 master.cpp:3853] Authenticating scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:28.051167 32257 sched.cpp:463] Scheduler::reregistered took 52801ns I1029 08:25:28.051723 32261 authenticator.hpp:161] Creating new server SASL connection I1029 08:25:28.052042 32249 authenticatee.hpp:224] Received SASL authentication mechanisms: CRAM-MD5 I1029 08:25:28.052077 32249 authenticatee.hpp:250] Attempting to authenticate with mechanism 'CRAM-MD5' I1029 08:25:28.052170 32249 master.cpp:1534] Dropping re-registration request of framework 20141029-082526-3142697795-40696-32232-0000 (default) at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 because new authentication attempt is in progress I1029 08:25:28.052218 32257 authenticator.hpp:267] Received SASL authentication start I1029 08:25:28.052325 32257 authenticator.hpp:389] Authentication requires more steps I1029 08:25:28.052428 32257 authenticatee.hpp:270] Received SASL authentication step I1029 08:25:28.052641 32246 authenticator.hpp:295] Received SASL authentication step I1029 08:25:28.052685 32246 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I1029 08:25:28.052701 32246 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I1029 08:25:28.052739 32246 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I1029 08:25:28.052767 32246 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I1029 08:25:28.052779 32246 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I1029 08:25:28.052788 32246 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I1029 08:25:28.052804 32246 authenticator.hpp:381] Authentication success I1029 08:25:28.052947 32252 authenticatee.hpp:310] Authentication success I1029 08:25:28.053020 32246 master.cpp:3893] Successfully authenticated principal 'test-principal' at scheduler-9ba6b803-40b4-48b9-bcef-45a329f6b2a4@67.195.81.187:40696 I1029 08:25:28.053462 32247 sched.cpp:357] Successfully authenticated with master master@67.195.81.187:40696 I1029 08:25:29.046855 32261 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:29.046880 32261 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 35632ns I1029 08:25:30.047458 32253 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:30.047487 32253 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 43031ns I1029 08:25:31.028373 32261 master.cpp:120] No whitelist given. Advertising offers for all slaves I1029 08:25:31.048673 32249 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:31.048702 32249 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 44769ns I1029 08:25:32.049576 32259 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:32.049604 32259 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 51919ns I1029 08:25:33.050864 32249 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:33.050896 32249 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 38019ns I1029 08:25:34.051961 32251 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:34.051993 32251 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 64619ns I1029 08:25:35.052196 32249 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:35.052223 32249 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 34475ns I1029 08:25:36.029101 32259 master.cpp:120] No whitelist given. Advertising offers for all slaves I1029 08:25:36.053067 32249 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:36.053095 32249 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 38354ns I1029 08:25:37.053506 32259 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1029 08:25:37.053536 32259 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 38249ns tests/master_authorization_tests.cpp:877: Failure Failed to wait 10secs for frameworkReregisteredMessage I1029 08:25:38.053241 32259 master.cpp:768] Framework 2014... | 2 |
| 159 | MESOS-2017 | Segfault with "Pure virtual method called" when tests fail | The most recent one: {noformat:title=DRFAllocatorTest.DRFAllocatorProcess} [ RUN ] DRFAllocatorTest.DRFAllocatorProcess Using temporary directory '/tmp/DRFAllocatorTest_DRFAllocatorProcess_BI905j' I1030 05:55:06.934813 24459 leveldb.cpp:176] Opened db in 3.175202ms I1030 05:55:06.935925 24459 leveldb.cpp:183] Compacted db in 1.077924ms I1030 05:55:06.935976 24459 leveldb.cpp:198] Created db iterator in 16460ns I1030 05:55:06.935995 24459 leveldb.cpp:204] Seeked to beginning of db in 2018ns I1030 05:55:06.936005 24459 leveldb.cpp:273] Iterated through 0 keys in the db in 335ns I1030 05:55:06.936039 24459 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I1030 05:55:06.936705 24480 recover.cpp:437] Starting replica recovery I1030 05:55:06.937023 24480 recover.cpp:463] Replica is in EMPTY status I1030 05:55:06.938158 24475 replica.cpp:638] Replica in EMPTY status received a broadcasted recover request I1030 05:55:06.938859 24482 recover.cpp:188] Received a recover response from a replica in EMPTY status I1030 05:55:06.939486 24474 recover.cpp:554] Updating replica status to STARTING I1030 05:55:06.940249 24489 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 591981ns I1030 05:55:06.940274 24489 replica.cpp:320] Persisted replica status to STARTING I1030 05:55:06.940752 24481 recover.cpp:463] Replica is in STARTING status I1030 05:55:06.940820 24489 master.cpp:312] Master 20141030-055506-3142697795-40429-24459 (pomona.apache.org) started on 67.195.81.187:40429 I1030 05:55:06.940871 24489 master.cpp:358] Master only allowing authenticated frameworks to register I1030 05:55:06.940891 24489 master.cpp:363] Master only allowing authenticated slaves to register I1030 05:55:06.940908 24489 credentials.hpp:36] Loading credentials for authentication from '/tmp/DRFAllocatorTest_DRFAllocatorProcess_BI905j/credentials' I1030 05:55:06.941215 24489 master.cpp:392] Authorization enabled I1030 05:55:06.941751 24475 master.cpp:120] No whitelist given. Advertising offers for all slaves I1030 05:55:06.942227 24474 replica.cpp:638] Replica in STARTING status received a broadcasted recover request I1030 05:55:06.942401 24476 hierarchical_allocator_process.hpp:299] Initializing hierarchical allocator process with master : master@67.195.81.187:40429 I1030 05:55:06.942895 24483 recover.cpp:188] Received a recover response from a replica in STARTING status I1030 05:55:06.943035 24474 master.cpp:1242] The newly elected leader is master@67.195.81.187:40429 with id 20141030-055506-3142697795-40429-24459 I1030 05:55:06.943063 24474 master.cpp:1255] Elected as the leading master! I1030 05:55:06.943079 24474 master.cpp:1073] Recovering from registrar I1030 05:55:06.943313 24480 registrar.cpp:313] Recovering registrar I1030 05:55:06.943455 24475 recover.cpp:554] Updating replica status to VOTING I1030 05:55:06.944144 24474 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 536365ns I1030 05:55:06.944172 24474 replica.cpp:320] Persisted replica status to VOTING I1030 05:55:06.944355 24489 recover.cpp:568] Successfully joined the Paxos group I1030 05:55:06.944576 24489 recover.cpp:452] Recover process terminated I1030 05:55:06.945155 24486 log.cpp:656] Attempting to start the writer I1030 05:55:06.947013 24473 replica.cpp:474] Replica received implicit promise request with proposal 1 I1030 05:55:06.947854 24473 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 806463ns I1030 05:55:06.947883 24473 replica.cpp:342] Persisted promised to 1 I1030 05:55:06.948547 24481 coordinator.cpp:230] Coordinator attemping to fill missing position I1030 05:55:06.950269 24479 replica.cpp:375] Replica received explicit promise request for position 0 with proposal 2 I1030 05:55:06.950933 24479 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 603843ns I1030 05:55:06.950961 24479 replica.cpp:676] Persisted action at 0 I1030 05:55:06.952180 24476 replica.cpp:508] Replica received write request for position 0 I1030 05:55:06.952239 24476 leveldb.cpp:438] Reading position from leveldb took 28437ns I1030 05:55:06.952896 24476 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 623980ns I1030 05:55:06.952926 24476 replica.cpp:676] Persisted action at 0 I1030 05:55:06.953543 24485 replica.cpp:655] Replica received learned notice for position 0 I1030 05:55:06.954082 24485 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 511807ns I1030 05:55:06.954107 24485 replica.cpp:676] Persisted action at 0 I1030 05:55:06.954128 24485 replica.cpp:661] Replica learned NOP action at position 0 I1030 05:55:06.954710 24473 log.cpp:672] Writer started with ending position 0 I1030 05:55:06.956215 24478 leveldb.cpp:438] Reading position from leveldb took 33085ns I1030 05:55:06.959481 24475 registrar.cpp:346] Successfully fetched the registry (0B) in 16.11904ms I1030 05:55:06.959616 24475 registrar.cpp:445] Applied 1 operations in 28239ns; attempting to update the 'registry' I1030 05:55:06.962514 24487 log.cpp:680] Attempting to append 139 bytes to the log I1030 05:55:06.962646 24474 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I1030 05:55:06.964146 24486 replica.cpp:508] Replica received write request for position 1 I1030 05:55:06.964962 24486 leveldb.cpp:343] Persisting action (158 bytes) to leveldb took 743389ns I1030 05:55:06.964993 24486 replica.cpp:676] Persisted action at 1 I1030 05:55:06.965895 24473 replica.cpp:655] Replica received learned notice for position 1 I1030 05:55:06.966531 24473 leveldb.cpp:343] Persisting action (160 bytes) to leveldb took 607242ns I1030 05:55:06.966555 24473 replica.cpp:676] Persisted action at 1 I1030 05:55:06.966578 24473 replica.cpp:661] Replica learned APPEND action at position 1 I1030 05:55:06.967706 24481 registrar.cpp:490] Successfully updated the 'registry' in 8.036096ms I1030 05:55:06.967895 24481 registrar.cpp:376] Successfully recovered registrar I1030 05:55:06.967993 24482 log.cpp:699] Attempting to truncate the log to 1 I1030 05:55:06.968258 24479 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I1030 05:55:06.968268 24475 master.cpp:1100] Recovered 0 slaves from the Registry (101B) ; allowing 10mins for slaves to re-register I1030 05:55:06.969156 24476 replica.cpp:508] Replica received write request for position 2 I1030 05:55:06.969678 24476 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 491913ns I1030 05:55:06.969703 24476 replica.cpp:676] Persisted action at 2 I1030 05:55:06.970459 24478 replica.cpp:655] Replica received learned notice for position 2 I1030 05:55:06.971060 24478 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 573076ns I1030 05:55:06.971124 24478 leveldb.cpp:401] Deleting ~1 keys from leveldb took 35339ns I1030 05:55:06.971145 24478 replica.cpp:676] Persisted action at 2 I1030 05:55:06.971168 24478 replica.cpp:661] Replica learned TRUNCATE action at position 2 I1030 05:55:06.980211 24459 containerizer.cpp:100] Using isolation: posix/cpu,posix/mem I1030 05:55:06.984153 24473 slave.cpp:169] Slave started on 203)@67.195.81.187:40429 I1030 05:55:07.055308 24473 credentials.hpp:84] Loading credential for authentication from '/tmp/DRFAllocatorTest_DRFAllocatorProcess_wULx31/credential' I1030 05:55:06.988750 24459 sched.cpp:137] Version: 0.21.0 I1030 05:55:07.055521 24473 slave.cpp:276] Slave using credential for: test-principal I1030 05:55:07.055726 24473 slave.cpp:289] Slave resources: cpus(*):2; mem(*):1024; disk(*):0; ports(*):[31000-32000] I1030 05:55:07.055865 24473 slave.cpp:318] Slave hostname: pomona.apache.org I1030 05:55:07.055881 24473 slave.cpp:319] Slave checkpoint: false W1030 05:55:07.055889 24473 slave.cpp:321] Disabling checkpointing is deprecated and the --checkpoint flag will be removed in a future release. Please avoid using this flag I1030 05:55:07.056172 24485 sched.cpp:233] New master detected at master@67.195.81.187:40429 I1030 05:55:07.056222 24485 sched.cpp:283] Authenticating with master master@67.195.81.187:40429 I1030 05:55:07.056717 24485 state.cpp:33] Recovering state from '/tmp/DRFAllocatorTest_DRFAllocatorProcess_wULx31/meta' I1030 05:55:07.056851 24475 authenticatee.hpp:133] Creating new client SASL connection I1030 05:55:07.057003 24473 status_update_manager.cpp:197] Recovering status update manager I1030 05:55:07.057252 24488 master.cpp:3853] Authenticating scheduler-c98e7aac-d03f-464a-aa75-61208600e196@67.195.81.187:40429 I1030 05:55:07.057502 24489 containerizer.cpp:281] Recovering containerizer I1030 05:55:07.057524 24475 authenticator.hpp:161] Creating new server SASL connection I1030 05:55:07.057688 24475 authenticatee.hpp:224] Received SASL authentication mechanisms: CRAM-MD5 I1030 05:55:07.057719 24475 authenticatee.hpp:250] Attempting to authenticate with mechanism 'CRAM-MD5' I1030 05:55:07.057919 24481 authenticator.hpp:267] Received SASL authentication start I1030 05:55:07.057968 24481 authenticator.hpp:389] Authentication requires more steps I1030 05:55:07.058070 24473 authenticatee.hpp:270] Received SASL authentication step I1030 05:55:07.058199 24485 authenticator.hpp:295] Received SASL authentication step I1030 05:55:07.058223 24485 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I1030 05:55:07.058233 24485 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I1030 05:55:07.058259 24485 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I1030 05:55:07.058290 24485 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I1030 05:55:07.058302 24485 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I1030 05:55:07.058307 24485 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I1030 05:55:07.058320 24485 authenticator.hpp:381] Authentication success I1030 05:55:07.058467 24480 master.cpp:3893] Successfully authenticated principal 'test-principal' at scheduler-c98e7aac-d03f-464a-aa75-61208600e196@67.195.81.187:40429 I1030 05:55:07.058493 24485 slave.cpp:3456] Finished recovery I1030 05:55:07.058593 24478 authenticatee.hpp:310] Authentication success I1030 05:55:07.058838 24478 sched.cpp:357] Successfully authenticated with master master@67.195.81.187:40429 I1030 05:55:07.058861 24478 sched.cpp:476] Sending registration request to master@67.195.81.187:40429 I1030 05:55:07.058969 24475 slave.cpp:602] New master detected at master@67.195.81.187:40429 I1030 05:55:07.058969 24487 status_update_manager.cpp:171] Pausing sending status updates I1030 05:55:07.059026 24475 slave.cpp:665] Authenticating with master master@67.195.81.187:40429 I1030 05:55:07.059061 24481 master.cpp:1362] Received registration request for framework 'framework1' at scheduler-c98e7aac-d03f-464a-aa75-61208600e196@67.195.81.187:40429 I1030 05:55:07.059131 24481 master.cpp:1321] Authorizing framework principal 'test-principal' to receive offers for role 'role1' I1030 05:55:07.059171 24475 slave.cpp:638] Detecting new master I1030 05:55:07.059214 24482 authenticatee.hpp:133] Creating new client SASL connection I1030 05:55:07.059550 24481 master.cpp:3853] Authenticating slave(203)@67.195.81.187:40429 I1030 05:55:07.059787 24487 authenticator.hpp:161] Creating new server SASL connection I1030 05:55:07.059922 24481 master.cpp:1426] Registering framework 20141030-055506-3142697795-40429-24459-0000 (framework1) at scheduler-c98e7aac-d03f-464a-aa75-61208600e196@67.195.81.187:40429 I1030 05:55:07.059996 24474 authenticatee.hpp:224] Received SASL authentication mechanisms: CRAM-MD5 I1030 05:55:07.060034 24474 authenticatee.hpp:250] Attempting to authenticate with mechanism 'CRAM-MD5' I1030 05:55:07.060117 24474 authenticator.hpp:267] Received SASL authentication start I1030 05:55:07.060165 24474 authenticator.hpp:389] Authentication requires more steps I1030 05:55:07.060377 24476 hierarchical_allocator_process.hpp:329] Added framework 20141030-055506-3142697795-40429-24459-0000 I1030 05:55:07.060394 24488 sched.cpp:407] Framework registered with 20141030-055506-3142697795-40429-24459-0000 I1030 05:55:07.060403 24476 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1030 05:55:07.060431 24476 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 29857ns I1030 05:55:07.060443 24488 sched.cpp:421] Scheduler::registered took 19407ns I1030 05:55:07.060545 24478 authenticatee.hpp:270] Received SASL authentication step I1030 05:55:07.060645 24478 authenticator.hpp:295] Received SASL authentication step I1030 05:55:07.060673 24478 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I1030 05:55:07.060685 24478 auxprop.cpp:153] Looking up auxiliary property '*userPassword' I1030 05:55:07.060714 24478 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I1030 05:55:07.060740 24478 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'pomona.apache.org' server FQDN: 'pomona.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I1030 05:55:07.060760 24478 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I1030 05:55:07.060770 24478 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I1030 05:55:07.060788 24478 authenticator.hpp:381] Authentication success I1030 05:55:07.060920 24474 authenticatee.hpp:310] Authentication success I1030 05:55:07.060945 24485 master.cpp:3893] Successfully authenticated principal 'test-principal' at slave(203)@67.195.81.187:40429 I1030 05:55:07.061388 24489 slave.cpp:722] Successfully authenticated with master master@67.195.81.187:40429 I1030 05:55:07.061504 24489 slave.cpp:1050] Will retry registration in 4.778336ms if necessary I1030 05:55:07.061718 24480 master.cpp:3032] Registering slave at slave(203)@67.195.81.187:40429 (pomona.apache.org) with id 20141030-055506-3142697795-40429-24459-S0 I1030 05:55:07.062119 24489 registrar.cpp:445] Applied 1 operations in 53691ns; attempting to update the 'registry' I1030 05:55:07.065182 24479 log.cpp:680] Attempting to append 316 bytes to the log I1030 05:55:07.065337 24487 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I1030 05:55:07.066359 24474 replica.cpp:508] Replica received write request for position 3 I1030 05:55:07.066643 24474 leveldb.cpp:343] Persisting action (335 bytes) to leveldb took 249579ns I1030 05:55:07.066671 24474 replica.cpp:676] Persisted action at 3 I../../src/tests/allocator_tests.cpp:120: Failure Failed to wait 10secs for offers1 1030 05:55:07.067101 24477 slave.cpp:1050] Will retry registration in 24.08243ms if necessary I1030 05:55:07.067140 24473 master.cpp:3020] Ignoring register slave message from slave(203)@67.195.81.187:40429 (pomona.apache.org) as admission is already in progress I1030 05:55:07.067395 24488 replica.cpp:655] Replica received learned notice for position 3 I1030 05:55:07.943416 24478 hierarchical_allocator_process.hpp:697] No resources available to allocate! I1030 05:55:19.804687 24478 hierarchical_allocator_process.hpp:659] Performed allocation for 0 slaves in 11.861261123secs I1030 05:55:11.942713 24474 master.cpp:120] No whitelist given. Advertising offers for all slaves I1030 05:55:19.805850 24488 leveldb.cpp:343] Persisting action (337 bytes) to leveldb took 1.067224ms I1030 05:55:19.806012 24488 replica.cpp:676] Persisted action at 3 ../../src/tests/allocator_tests.cpp:115: Failure Actual function call count doesn't match EXPECT_CALL(sched1, resourceOffers(_, _))... Expected: to be called once Actual: never called - unsatisfied and active I1030 05:55:19.806144 24488 replica.cpp:661] Replica learned APPEND action at position 3 I1030 05:55:19.806695 24473 master.cpp:768] Framework 20141030-055506-3142697795-40429-24459-0000 (framework1) at scheduler-c98e7aac-d03f-464a-aa75-61208600e196@67.195.81.187:40429 disconnected I1030 05:55:19.806726 24473 master.cpp:1731] Disconnecting framework 20141030-055506-3142697795-40429-24459-0000 (framework1) at scheduler-c98e7aac-d03f-464a-aa75-61208600e196@67.195.81.187:40429 I1030 05:55:19.806751 24473 master.cpp:1747] Deactivating framework 20141030-055506-3142697795-40429-24459-0000 (framework1) at scheduler-c98e7aac-d03f-464a-aa75-61208600e196@67.195.81.187:40429 I1030 05:55:19.806967 24473 master.cpp:790] Giving framework 20141030-055506-3142697795-40429-24459-0000 (framework1) at scheduler-c98e7aac-d03f-464a-aa75-61208600e196@67.195.81.187:40429 0ns to failover ../../src/tests/allocator_tests.cpp:94: Failure Actual function call count doesn't match EXPECT_CALL(allocator, slaveAdded(_, _, _))... Expected: to be called once Actual: never called - unsatisfied and active F1030 05:55:19.806967 24480 logging.cpp:57] RAW: Pure virtual method called I1030 05:55:19.807348 24488 master.cpp:3665] Framework failover timeout, removing framework 20141030-055506-3142697795-40429-24459-0000 (framework1) at scheduler-c98e7aac-d03f-464a-aa75-61208600e196@67.195.81.187:40429 I1030 05:55:19.807370 24488 master.cpp:4201] Removing framework 20141030-055506-3142697795-40429-24459-0000 (framework1) at scheduler-c98e7aac-d03f-464a-aa75-61208600e196@67.195.81.187:40429 *** Aborted at 1414648519 (unix time) try "date -d @1414648519" if you are using GNU date *** PC: @ 0x91bc86 process::PID<>::PID() *** SIGSEGV (@0x0) received by PID 24459 (TID 0x2b86c919a700) from PID 0; stack trace: *** I1030 05:55:19.808631 24489 registrar.cpp:490] Successfully updated the 'registry' in 12.746377984secs @ 0x2b86c55fc340 (unknown) I1030 05:55:19.808938 24473 log.cpp:699] Attempting to truncate the log to 3 @ 0x2b86c3327174 google::LogMessage::Fail() I1030 05:55:19.809084 24481 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 @ 0x91bc86 process::PID<>::PID() @ 0x2b86c332c868 google::RawLog__() I1030 05:55:19.810191 24479 replica.cpp:508] Replica received write request for position 4 I1030 05:55:19.810899 24479 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 678090ns I1030 05:55:19.810919 24479 replica.cpp:676] Persisted action at 4 @ 0x91bf24 process::Process<>::self() I1030 05:55:19.811635 24485 replica.cpp:655] Replica received learned notice for position 4 I1030 05:55:19.812180 24485 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 523927ns I1030 05:55:19.812228 24485 leveldb.cpp:401] Deleting ~2 keys from leveldb took 29523ns I1030 05:55:19.812242 24485 replica.cpp:676] Persisted action at 4 I @ 0x2b86c29d2a36 __cxa_pure_virtual 1030 05:55:19.812258 24485 replica.cpp:661] Replica learned TRUNCATE action at position 4 @ 0x1046936 testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith() I1030 05:55:19.829655 24474 slave.cpp:1050] Will retry registration in 31.785967ms if necessary @ 0x9c0633 testing::internal::FunctionMockerBase<>::InvokeWith() @ 0x9b6152 testing::internal::FunctionMocker<>::Invoke() @ 0x9abdeb mesos::internal::tests::MockAllocatorPr... | 5 |
| 160 | MESOS-2029 | Allow slave to checkpoint resources. | The checkpointed resources are independent of the slave lifecycle. In other words, even if the slave host reboots, it'll still recover the checkpointed resources (unlike other checkpointed data). The slave needs to verify during startup that the checkpointed resources are compatible with the resources of the slave (specified using --resources flag). | 5 |
| 161 | MESOS-2030 | Maintain persistent disk resources in master memory. | Maintain an in-memory data structure to track persistent disk resources on each slave. Update this data structure when slaves register/re-register/disconnect, etc. | 3 |
| 162 | MESOS-2031 | Manage persistent directories on slave. | Whenever a slave sees a persistent disk resource (in ExecutorInfo or TaskInfo) that is new to it, it will create a persistent directory which is for tasks to store persistent data. The slave needs to do the following after it's created: 1) symlink into the executor sandbox so that tasks/executor can see it 2) garbage collect it once it is released by the framework | 5 |
| 163 | MESOS-2032 | Update Maintenance design to account for persistent resources. | With persistent resources and dynamic reservations, frameworks need to know how long the resources will be unavailable for maintenance operations. This is because for persistent resources, the framework needs to understand how long the persistent resource will be unavailable. For example, if there will be a 10 minute reboot for a kernel upgrade, the framework will not want to re-replicate all of it's persistent data on the machine. Rather, tolerating one unavailable replica for the maintenance window would be preferred. I'd like to do a revisit of the design to ensure it works well for persistent resources as well. | 13 |
| 164 | MESOS-2033 | Documentation for isolator filesystem/shared. | NULL | 1 |
| 165 | MESOS-2034 | Documentation for isolator namespaces/pid. | NULL | 1 |
| 166 | MESOS-2035 | Add reason to containerizer proto Termination | When an isolator kills a task, the reason is unknown. As part of MESOS-1830, the reason is set to a general one but ideally we would have the termination reason to pass through to the status update. | 5 |
| 167 | MESOS-2043 | Framework auth fail with timeout error and never get authenticated | I'm facing this issue in master as of https://github.com/apache/mesos/commit/74ea59e144d131814c66972fb0cc14784d3503d4 As [~adam-mesos] mentioned in IRC, this sounds similar to MESOS-1866. I'm running 1 master and 1 scheduler (aurora). The framework authentication fail due to time out: error on mesos master: {code} I1104 19:37:17.741449 8329 master.cpp:3874] Authenticating scheduler-d2d4437b-d375-4467-a583-362152fe065a@SCHEDULER_IP:8083 I1104 19:37:17.741585 8329 master.cpp:3885] Using default CRAM-MD5 authenticator I1104 19:37:17.742106 8336 authenticator.hpp:169] Creating new server SASL connection W1104 19:37:22.742959 8329 master.cpp:3953] Authentication timed out W1104 19:37:22.743548 8329 master.cpp:3930] Failed to authenticate scheduler-d2d4437b-d375-4467-a583-362152fe065a@SCHEDULER_IP:8083: Authentication discarded {code} scheduler error: {code} I1104 19:38:57.885486 49012 sched.cpp:283] Authenticating with master master@MASTER_IP:PORT I1104 19:38:57.885928 49002 authenticatee.hpp:133] Creating new client SASL connection I1104 19:38:57.890581 49007 authenticatee.hpp:224] Received SASL authentication mechanisms: CRAM-MD5 I1104 19:38:57.890656 49007 authenticatee.hpp:250] Attempting to authenticate with mechanism 'CRAM-MD5' W1104 19:39:02.891196 49005 sched.cpp:378] Authentication timed out I1104 19:39:02.891850 49018 sched.cpp:338] Failed to authenticate with master master@MASTER_IP:PORT: Authentication discarded {code} Looks like 2 instances {{scheduler-20f88a53-5945-4977-b5af-28f6c52d3c94}} & {{scheduler-d2d4437b-d375-4467-a583-362152fe065a}} of same framework is trying to authenticate and fail. {code} W1104 19:36:30.769420 8319 master.cpp:3930] Failed to authenticate scheduler-20f88a53-5945-4977-b5af-28f6c52d3c94@SCHEDULER_IP:8083: Failed to communicate with authenticatee I1104 19:36:42.701441 8328 master.cpp:3860] Queuing up authentication request from scheduler-d2d4437b-d375-4467-a583-362152fe065a@SCHEDULER_IP:8083 because authentication is still in progress {code} Restarting master and scheduler didn't fix it. This particular issue happen with 1 master and 1 scheduler after MESOS-1866 is fixed. | 5 |
| 168 | MESOS-2044 | Use one IP address per container for network isolation | If there are enough IP addresses, either IPv4 or IPv6, we should use one IP address per container, instead of the ugly port range based solution. One problem with this is the IP address management, usually it is managed by a DHCP server, maybe we need to manage them in mesos master/slave. Also, maybe use macvlan instead of veth for better isolation. | 40 |
| 169 | MESOS-2051 | Pull Metrics struct out of Master and Slave to improve readability | NULL | 2 |
| 170 | MESOS-2052 | RunState::recover should always recover 'completed' | RunState::recover() will return partial state if it cannot find or open the libprocess pid file. Specifically, it does not recover the 'completed' flag. However, if the slave has removed the executor (because launch failed or the executor failed to register) the sentinel flag will be set and this fact should be recovered. This ensures that container recovery is not attempted later. This was discovered when the LinuxLauncher failed to recover because it was asked to recover two containers with the same forkedPid. Investigation showed the executors both OOM'ed before registering, i.e., no libprocess pid file was present. However, the containerizer had detected the OOM, destroyed the container, and notified the slave which cleaned everything up: failing the task and calling removeExecutor (which writes the completed sentinel file.) | 1 |
| 171 | MESOS-2055 | MesosContainerizerExecuteTest.IoRedirection test is flaky | Observed this on ASF CI: {code} [ RUN ] MesosContainerizerExecuteTest.IoRedirection Using temporary directory '/tmp/MesosContainerizerExecuteTest_IoRedirection_PbBn8a' I1108 00:34:25.820514 30391 containerizer.cpp:100] Using isolation: posix/cpu,posix/mem I1108 00:34:25.821048 30411 containerizer.cpp:424] Starting container 'test_container' for executor 'executor' of framework '' I1108 00:34:25.824015 30411 launcher.cpp:137] Forked child with pid '4221' for container 'test_container' I1108 00:34:25.825438 30408 containerizer.cpp:571] Fetching URIs for container 'test_container' using command '/home/jenkins/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/build/src/mesos-fetcher' I1108 00:34:25.984254 30419 containerizer.cpp:1117] Executor for container 'test_container' has exited I1108 00:34:25.984341 30419 containerizer.cpp:946] Destroying container 'test_container' ../../src/tests/containerizer_tests.cpp:487: Failure Value of: (os::read(path::join(directory, "stderr"))).get() Actual: "I1108 00:34:25.872990 4224 logging.cpp:177] Logging to STDERR\nthis is stderr\n" Expected: errMsg + "\n" Which is: "this is stderr\n" [ FAILED ] MesosContainerizerExecuteTest.IoRedirection (185 ms) [----------] 1 test from MesosContainerizerExecuteTest (185 ms total) {code} | 1 |
| 172 | MESOS-2056 | Refactor fetcher code in preparation for fetcher cache | Refactor/rearrange fetcher-related code so that cache functionality can be dropped in. One could do both together in one go. This is splitting up reviews into smaller chunks. It will not immediately be obvious how this change will be used later, but it will look better-factored and still do the exact same thing as before. In particular, a download routine to be reused several times in launcher/fetcher will be factored out and the remainder of fetcher-related code can be moved from the containerizer realm into fetcher.cpp. | 1 |
| 173 | MESOS-2057 | Concurrency control for fetcher cache | Having added a URI flag to CommandInfo messages (in MESOS-2069) that indicates caching, caching files downloaded by the fetcher in a repository, now ensure that when a URI is "cached", it is only ever downloaded once for the same user on the same slave as long as the slave keeps running. This even holds if multiple tasks request the same URI concurrently. If multiple requests for the same URI occur, perform only one of them and reuse the result. Make concurrent requests for the same URI wait for the one download. Different URIs from different CommandInfos can be downloaded concurrently. No cache eviction, cleanup or failover will be handled for now. Additional tickets will be filed for these enhancements. (So don't use this feature in production until the whole epic is complete.) Note that implementing this does not suffice for production use. This ticket contains the main part of the fetcher logic, though. See the epic MESOS-336 for the rest of the features that lead to a fully functional fetcher cache. The proposed general approach is to keep all bookkeeping about what is in which stage of being fetched and where it resides in the slave's MesosContainerizerProcess, so that all concurrent access is disambiguated and controlled by an "actor" (aka libprocess "process"). Depends on MESOS-2056 and MESOS-2069. | 8 |
| 174 | MESOS-2058 | Deprecate stats.json endpoints for Master and Slave | With the introduction of the libprocess {{/metrics/snapshot}} endpoint, metrics are now duplicated in the Master and Slave between this and {{stats.json}}. We should deprecate the {{stats.json}} endpoints. Manual inspection of {{stats.json}} shows that all metrics are now covered by the new endpoint for Master and Slave. | 1 |
| 175 | MESOS-2061 | Add InverseOffer protobuf message. | InverseOffer was defined as part of the maintenance work in MESOS-1474, design doc here: https://docs.google.com/document/d/16k0lVwpSGVOyxPSyXKmGC-gbNmRlisNEe4p-fAUSojk/edit?usp=sharing {code} /** * A request to return some resources occupied by a framework. */ message InverseOffer { required OfferID id = 1; required FrameworkID framework_id = 2; // A list of resources being requested back from the framework. repeated Resource resources = 3; // Specified if the resources need to be released from a particular slave. optional SlaveID slave_id = 4; // The resources in this InverseOffer are part of a planned maintenance // schedule in the specified window. Any tasks running using these // resources may be killed when the window arrives. optional Interval unavailability = 5; } {code} This ticket is to capture the addition of the InverseOffer protobuf to mesos.proto, the necessary API changes for Event/Call and the language bindings will be tracked separately. | 3 |
| 176 | MESOS-2062 | Add InverseOffer to Event/Call API. | The initial use case for InverseOffer in the framework API will be the maintenance primitives in mesos: MESOS-1474. One way to add this is to tack it on to the OFFERS Event: {code} message Offers { repeated Offer offers = 1; repeated InverseOffer inverse_offers = 2; } {code} | 3 |
| 177 | MESOS-2063 | Add InverseOffer to C++ Scheduler API. | The initial use case for InverseOffer in the framework API will be the maintenance primitives in mesos: MESOS-1474. One way to add these to the C++ Scheduler API is to add a new callback: {code} virtual void inverseResourceOffers( SchedulerDriver* driver, const std::vector<InverseOffer>& inverseOffers) = 0; {code} libmesos compatibility will need to be figured out here. We may want to leave the C++ binding untouched in favor of Event/Call, in order to not break API compatibility for schedulers. | 5 |
| 178 | MESOS-2064 | Add InverseOffer to Java Scheduler API. | The initial use case for InverseOffer in the framework API will be the maintenance primitives in mesos: MESOS-1474. One way to add these to the Java Scheduler API is to add a new callback: {code} void inverseResourceOffers( SchedulerDriver driver, List<InverseOffer> inverseOffers); {code} JAR / libmesos compatibility will need to be figured out here. We may want to leave the Java binding untouched in favor of Event/Call, in order to not break API compatibility for schedulers. | 5 |
| 179 | MESOS-2065 | Add InverseOffer to Python Scheduler API. | The initial use case for InverseOffer in the framework API will be the maintenance primitives in mesos: MESOS-1474. One way to add these to the Python Scheduler API is to add a new callback: {code} def inverseResourceOffers(self, driver, inverse_offers): {code} Egg / libmesos compatibility will need to be figured out here. We may want to leave the Python binding untouched in favor of Event/Call, in order to not break API compatibility for schedulers. | 5 |
| 180 | MESOS-2066 | Add optional 'Unavailability' to resource offers to provide maintenance awareness. | In order to inform frameworks about upcoming maintenance on offered resources, per MESOS-1474, we'd like to add an optional 'Unavailability' information to offers: {code} message Interval { optional double start = 1; // Time, in seconds since the Epoch. optional double duration = 2; // Time, in seconds. } message Offer { // Existing fields ... // Signifies that the resources in this Offer are part of a planned // maintenance schedule in the specified window. Any tasks launched // using these resources may be killed when the window arrives. // This field gives additional information about the maintenance. // The maintenance may not necessarily start at exactly at this interval, // nor last for exactly the duration of this interval. optional Interval unavailability = 9; } {code} | 3 |
| 181 | MESOS-2067 | Add HTTP API to the master for maintenance operations. | Based on MESOS-1474, we'd like to provide an HTTP API on the master for the maintenance primitives in mesos. For the MVP, we'll want something like this for manipulating the schedule: {code} /maintenance/schedule GET - returns the schedule, which will include the various maintenance windows. POST - create or update the schedule with a JSON blob (see below). /maintenance/status GET - returns a list of machines and their maintenance mode. /maintenance/start POST - Transition a set of machines from Draining into Deactivated mode. /maintenance/stop POST - Transition a set of machines from Deactivated into Normal mode. /maintenance/consensus <- (Not sure what the right name is. matrix? acceptance?) GET - Returns the latest info on which frameworks have accepted or declined the maintenance schedule. {code} (Note: The slashes in URLs might not be supported yet.) A schedule might look like: {code} { "windows" : [ { "machines" : [ { "ip" : "192.168.0.1" }, { "hostname" : "localhost" }, ... ], "unavailability" : { "start" : 12345, // Epoch seconds. "duration" : 1000 // Seconds. } }, ... ] } {code} There should be firewall settings such that only those with access to master can use these endpoints. | 8 |
| 182 | MESOS-2069 | Basic fetcher cache functionality | Add a flag to CommandInfo URI protobufs that indicates that files downloaded by the fetcher shall be cached in a repository. To be followed by MESOS-2057 for concurrency control. Also see MESOS-336 for the overall goals for the fetcher cache. | 8 |
| 183 | MESOS-2070 | Implement simple slave recovery behavior for fetcher cache | Clean the fetcher cache completely upon slave restart/recovery. This implements correct, albeit not ideal behavior. More efficient schemes that restore knowledge about cached files or even resume downloads can be added later. | 2 |
| 184 | MESOS-2072 | Fetcher cache eviction | Delete files from the fetcher cache so that a given cache size is never exceeded. Succeed in doing so while concurrent downloads are on their way and new requests are pouring in. Idea: measure the size of each download before it begins, make enough room before the download. This means that only download mechanisms that divulge the size before the main download will be supported. AFAWK, those in use so far have this property. The calculation of how much space to free needs to be under concurrency control, accumulating all space needed for competing, incomplete download requests. (The Python script that performs fetcher caching for Aurora does not seem to implement this. See https://gist.github.com/zmanji/f41df77510ef9d00265a, imagine several of these programs running concurrently, each one's _cache_eviction() call succeeding, each perceiving the SAME free space being available.) Ultimately, a conflict resolution strategy is needed if just the downloads underway already exceed the cache capacity. Then, as a fallback, direct download into the work directory will be used for some tasks. TBD how to pick which task gets treated how. At first, only support copying of any downloaded files to the work directory for task execution. This isolates the task life cycle after starting a task from cache eviction considerations. (Later, we can add symbolic links that avoid copying. But then eviction of fetched files used by ongoing tasks must be blocked, which adds complexity. another future extension is MESOS-1667 "Extract from URI while downloading into work dir"). | 8 |
| 185 | MESOS-2074 | Fetcher cache test fixture | To accelerate providing good test coverage for the fetcher cache (MESOS-336), we can provide a framework that canonicalizes creating and running a number of tasks and allows easy parametrization with combinations of the following: - whether to cache or not - whether make what has been downloaded executable or not - whether to extract from an archive or not - whether to download from a file system, http, or... We can create a simple HHTP server in the test fixture to support the latter. Furthermore, the tests need to be robust wrt. varying numbers of StatusUpdate messages. An accumulating update message sink that reports the final state is needed. All this has already been programmed in this patch, just needs to be rebased: https://reviews.apache.org/r/21316/ | 5 |
| 186 | MESOS-2075 | Add maintenance information to the replicated registry. | To achieve fault-tolerance for the maintenance primitives, we will need to add the maintenance information to the registry. The registry currently stores all of the slave information, which is quite large (~ 17MB for 50,000 slaves from my testing), which results in a protobuf object that is extremely expensive to copy. As far as I can tell, reads / writes to maintenance information is independent of reads / writes to the existing 'registry' information. So there are two approach here: h4. Add maintenance information to 'maintenance' key: # The advantage of this approach is that we don't further grow the large Registry object. # This approach assumes that writes to 'maintenance' are independent of writes to the 'registry'. -If these writes are not independent, this approach requires that we add transactional support to the State abstraction.- # -This approach requires adding compaction to LogStorage.- # This approach likely requires some refactoring to the Registrar. h4. Add maintenance information to 'registry' key: (This is the chosen method.) # The advantage of this approach is that it's the easiest to implement. # This will further grow the single 'registry' object, but doesn't preclude it being split apart in the future. # This approach may require using the diff support in LogStorage and/or adding compression support to LogStorage snapshots to deal with the increased size of the registry. | 13 |
| 187 | MESOS-2076 | Implement maintenance primitives in the Master. | The master will need to do a number of things to implement the maintenance primitives: # For machines that have a maintenance window: #* Disambiguate machines to agents. #* For unused resources, offers must be augmented with an Unavailability. #* For used resources, inverse offers must be sent. # For inverse offers: #* Filter them before sending them again. #* For declined inverse offers, do something with the reason (store or log). # Recover the maintenance information upon failover. Note: Some amount of this logic will need to be placed in the allocator. | 13 |
| 188 | MESOS-2077 | Ensure that TASK_LOSTs for a hard slave drain (SIGUSR1) include a Reason. | For maintenance, sometimes operators will force the drain of a slave (via SIGUSR1), when deemed safe (e.g. non-critical tasks running) and/or necessary (e.g. bad hardware). To eliminate alerting noise, we'd like to add a 'Reason' that expresses the forced drain of the slave, so that these are not considered to be a generic slave removal TASK_LOST. | 3 |
| 189 | MESOS-2078 | Scheduler driver may ACK status updates when the scheduler threw an exception | [~vinodkone] discovered that this can happen if the scheduler calls {{SchedulerDriver#stop}} before or while handling {{Scheduler#statusUpdate}}. In src/sched/sched.cpp: The driver invokes {{statusUpdate}} and later checks the {{aborted}} flag to determine whether to send an ACK. {code} void statusUpdate( const UPID& from, const StatusUpdate& update, const UPID& pid) { ... scheduler->statusUpdate(driver, status); VLOG(1) << "Scheduler::statusUpdate took " << stopwatch.elapsed(); // Note that we need to look at the volatile 'aborted' here to // so that we don't acknowledge the update if the driver was // aborted during the processing of the update. if (aborted) { VLOG(1) << "Not sending status update acknowledgment message because " << "the driver is aborted!"; return; } ... {code} In src/java/jni/org_apache_mesos_MesosSchedulerDriver.cpp: The {{statusUpdate}} implementation checks for an exception and invokes {{driver->abort()}}. {code} void JNIScheduler::statusUpdate(SchedulerDriver* driver, const TaskStatus& status) { jvm->AttachCurrentThread(JNIENV_CAST(&env), NULL); jclass clazz = env->GetObjectClass(jdriver); jfieldID scheduler = env->GetFieldID(clazz, "scheduler", "Lorg/apache/mesos/Scheduler;"); jobject jscheduler = env->GetObjectField(jdriver, scheduler); clazz = env->GetObjectClass(jscheduler); // scheduler.statusUpdate(driver, status); jmethodID statusUpdate = env->GetMethodID(clazz, "statusUpdate", "(Lorg/apache/mesos/SchedulerDriver;" "Lorg/apache/mesos/Protos$TaskStatus;)V"); jobject jstatus = convert<TaskStatus>(env, status); env->ExceptionClear(); env->CallVoidMethod(jscheduler, statusUpdate, jdriver, jstatus); if (env->ExceptionCheck()) { env->ExceptionDescribe(); env->ExceptionClear(); jvm->DetachCurrentThread(); driver->abort(); return; } jvm->DetachCurrentThread(); } {code} In src/sched/sched.cpp: The {{abort()}} implementation exits early if {{status != DRIVER_RUNNING}}, and *does not set the aborted flag*. {code} Status MesosSchedulerDriver::abort() { Lock lock(&mutex); if (status != DRIVER_RUNNING) { return status; } CHECK(process != NULL); // We set the volatile aborted to true here to prevent any further // messages from being processed in the SchedulerProcess. However, // if abort() is called from another thread as the SchedulerProcess, // there may be at most one additional message processed. // TODO(bmahler): Use an atomic boolean. process->aborted = true; // Dispatching here ensures that we still process the outstanding // requests *from* the scheduler, since those do proceed when // aborted is true. dispatch(process, &SchedulerProcess::abort); return status = DRIVER_ABORTED; } {code} As a result, the code will ACK despite an exception being thrown. | 3 |
| 190 | MESOS-2080 | Add master metrics for maintenance. | We'll need metrics in order to gain visibility into the maintenance functionality. This will also allow operators to add alerting on these metrics, in particular: # Number of scheduled hosts. # Number of active windows. # Number of expired windows. # Number of successful drains. # Number of failed drains. As an example of an alert guideline, we would want to know the number of expired windows as a gauge to ensure that it is not growing excessively. This allows alerting to catch when operators are not properly unscheduling maintenance once it is complete. | 3 |
| 191 | MESOS-2081 | Add safety constraints for maintenance primitives. | In order to ensure that the maintenance primitives can be used safely by operators, we want to put a few safety mechanisms in place. Some ideas from the [design doc|https://docs.google.com/a/twitter.com/document/d/16k0lVwpSGVOyxPSyXKmGC-gbNmRlisNEe4p-fAUSojk/]: # Prevent bad schedules from being constructed: schedules with more than x% overlap in slaves are rejected. # Prevent bad maintenance from proceeding unchecked: if x% of the slaves are not being unscheduled, or are not re-registering, cancel the schedule. These will likely be configurable via flags. | 8 |
| 192 | MESOS-2082 | Update the webui to include maintenance information. | The simplest thing here would probably be to include another tab in the header for maintenance information. We could also consider adding maintenance information inline to the slaves table. Depending on how this is done, the maintenance tab could actually be a subset of the slaves table; only those slaves for which there is maintenance information. | 5 |
| 193 | MESOS-2083 | Add documentation for maintenance primitives. | We should provide some guiding documentation around the upcoming maintenance primitives in Mesos. Specifically, we should ensure that general users, framework developers, and operators understand the notion of maintenance in Mesos. Some guidance and recommendations for the latter two audiences will be necessary. | 8 |
| 194 | MESOS-2085 | Add support encrypted and non-encrypted communication in parallel for cluster upgrade | During cluster upgrade from non-encrypted to encrypted communication, we need to support an interim where: 1) A master can have connections to both encrypted and non-encrypted slaves 2) A slave that supports encrypted communication connects to a master that has not yet been upgraded. 3) Frameworks are encrypted but the master has not been upgraded yet. 4) Master has been upgraded but frameworks haven't. 5) A slave process has upgraded but running executor processes haven't. | 13 |
| 195 | MESOS-2097 | Update Resource protobuf with DiskInfo | {noformat} message Resource { required string name = 1; required Value.Type type = 2; optional Value.Scalar scalar = 3; optional Value.Ranges ranges = 4; optional Value.Set set = 5; optional string role = 6 [default = "*"]; // Used for describing persistent disk resource. message DiskInfo { // A unique identifier for the persistent disk resource. The id // needs to be unique within a role for a slave. required string id = 1; // The volume mapping for the persistent disk resource. required Volume volume = 2; } optional DiskInfo disk = 8; } {noformat} | 1 |
| 196 | MESOS-2098 | Update task validation to be after task authorization. | So that we can simply the task validation because we no longer need to check with pendingTasks. | 3 |
| 197 | MESOS-2099 | Support acquiring/releasing resources with DiskInfo in allocator. | The allocator needs to be changed because the resources are changing while we acquiring or releasing persistent disk resources (resources with DiskInfo). For example, when we release a persistent disk resource, we are changing the release with DiskInfo to a resource with the DiskInfo. | 8 |
| 198 | MESOS-2100 | Implement master to slave protocol for persistent disk resources. | We need to do the following: 1) Slave needs to send persisted resources when registering (or re-registering). 2) Master needs to send total persisted resources to slave by either re-using RunTask/UpdateFrameworkInfo or introduce new type of messages (like UpdateResources). | 8 |
| 199 | MESOS-2101 | Add the persistent resources release primitive to the framework API | We are thinking about introducing a Release protobuf message which specifies persistent disk resources (w/ DiskInfo) to release. The Release message could be piggybacked on the Launch/Decline message. This probably will overlap with the dynamic reservation work (MESOS-2018). | 3 |
| 200 | MESOS-2103 | Expose number of processes and threads in a container | The CFS cpu statistics (cpus_nr_throttled, cpus_nr_periods, cpus_throttled_time) are difficult to interpret. 1) nr_throttled is the number of intervals where *any* throttling occurred 2) throttled_time is the aggregate time *across all runnable tasks* (tasks in the Linux sense). For example, in a typical 60 second sampling interval: nr_periods = 600, nr_throttled could be 60, i.e., 10% of intervals, but throttled_time could be much higher than (60/600) * 60 = 6 seconds if there is more than one task that is runnable but throttled. *Each* throttled task contributes to the total throttled time. Small test to demonstrate throttled_time > nr_periods * quota_interval: 5 x {{'openssl speed'}} running with quota=100ms: {noformat} cat cpu.stat && sleep 1 && cat cpu.stat nr_periods 3228 nr_throttled 1276 throttled_time 528843772540 nr_periods 3238 nr_throttled 1286 throttled_time 531668964667 {noformat} All 10 intervals throttled (100%) for total time of 2.8 seconds in 1 second ("more than 100%" of the time interval) It would be helpful to expose the number of processes and tasks in the container cgroup. This would be at a very coarse granularity but would give some guidance. | 2 |
| 201 | MESOS-2104 | Correct naming of cgroup memory statistics | mem_rss_bytes is *not* RSS but is the total memory usage (memory.usage_in_bytes) of the cgroup, including file cache etc. Actual RSS is reported as mem_anon_bytes. These, and others, should be consistently named. | 3 |
| 202 | MESOS-2108 | Add configure flag or environment variable to enable SSL/libevent Socket | NULL | 1 |
| 203 | MESOS-2110 | Configurable Ping Timeouts | After a series of ping-failures, the master considers the slave lost and calls shutdownSlave, requiring such a slave that reconnects to kill its tasks and re-register as a new slaveId. On the other side, after a similar timeout, the slave will consider the master lost and try to detect a new master. These timeouts are currently hardcoded constants (5 * 15s), which may not be well-suited for all scenarios. - Some clusters may tolerate a longer slave process restart period, and wouldn't want tasks to be killed upon reconnect. - Some clusters may have higher-latency networks (e.g. cross-datacenter, or for volunteer computing efforts), and would like to tolerate longer periods without communication. We should provide flags/mechanisms on the master to control its tolerance for non-communicative slaves, and (less importantly?) on the slave to tolerate missing masters. | 8 |
| 204 | MESOS-2119 | Add Socket tests | Add more Socket specific tests to get coverage while doing libev to libevent (w and wo SSL) move | 5 |
| 205 | MESOS-2123 | Document changes in C++ Resources API in CHANGELOG. | With the refactor introduced in MESOS-1974, we need to document those API changes in CHANGELOG. | 2 |
| 206 | MESOS-2127 | killTask() should perform reconciliation for unknown tasks. | Currently, {{killTask}} uses its own reconciliation logic, which has diverged from the {{reconcileTasks}} logic. Specifically, when the task is unknown and a non-strict registry is in use, {{killTask}} will not send TASK_LOST whereas {{reconcileTask}} will. We should make these consistent. | 3 |
| 207 | MESOS-2128 | Turning on cgroups_limit_swap effectively disables memory isolation | Our test runs show that enabling cgroups_limit_swap effectively disables memory isolation altogether. Per: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-memory.html "It is important to set the memory.limit_in_bytes parameter before setting the memory.memsw.limit_in_bytes parameter: attempting to do so in the reverse order results in an error. This is because memory.memsw.limit_in_bytes becomes available only after all memory limitations (previously set in memory.limit_in_bytes) are exhausted." Looks like the flag sets "memory.memsw.limit_in_bytes" if true and "memory.limit_in_bytes" if false, but should always set "memory.limit_in_bytes" and in addition set "memory.memsw.limit_in_bytes" if true. Otherwise the limits won't be set and enforced. See: https://github.com/apache/mesos/blob/c8598f7f5a24a01b6a68e0f060b79662ee97af89/src/slave/containerizer/isolators/cgroups/mem.cpp#L365 | 2 |
| 208 | MESOS-2135 | Support DiskInfo in C++ Resources | We need to change the following functions: 1) addable 2) subtractable 3) validate We probably shouldn't add two disk resources with the same persistence id because they must come from different "namespaces". We can add more checks in the validate functions (for protobufs). | 3 |
| 209 | MESOS-2136 | Expose per-cgroup memory pressure | The cgroup memory controller can provide information on the memory pressure of a cgroup. This is in the form of an event based notification where events of (low, medium, critical) are generated when the kernel makes specific actions to allocate memory. This signal is probably more informative than comparing memory usage to memory limit. | 5 |
| 210 | MESOS-2139 | Enable the master to handle reservation operations | master's {{_accept}} function currently only handles {{Create}} and {{Destroy}} operations which exist for persistent volumes. We need to handle the {{Reserve}} and {{Unreserve}} operations for dynamic reservations as well. In addition, we need to add {{validate}} functions for the reservation operations. | 5 |
| 211 | MESOS-2144 | Segmentation Fault in ExamplesTest.LowLevelSchedulerPthread | Occured on review bot review of: https://reviews.apache.org/r/28262/#review62333 The review doesn't touch code related to the test (And doesn't break libprocess in general) [ RUN ] ExamplesTest.LowLevelSchedulerPthread ../../src/tests/script.cpp:83: Failure Failed low_level_scheduler_pthread_test.sh terminated with signal Segmentation fault [ FAILED ] ExamplesTest.LowLevelSchedulerPthread (7561 ms) The test | 8 |
| 212 | MESOS-2147 | Large number of connections slows statistics.json responses. | We observed that in our production environment with network monitoring being turned on. If there are many connections (> 10^4) in a container, getting socket information is expensive. It might take 1min to process all the socket information. One of the reason is that the library we are using (libnl) is not so optimized. Cong Wang has already submitted a patch: http://lists.infradead.org/pipermail/libnl/2014-November/001715.html | 2 |
| 213 | MESOS-2157 | Add /master/slaves and /master/frameworks/{framework}/tasks/{task} endpoints | master/state.json exports the entire state of the cluster and can, for large clusters, become massive (tens of megabytes of JSON). Often, a client only need information about subsets of the entire state, for example all connected slaves, or information (registration info, tasks, etc) belonging to a particular framework. We can partition state.json into many smaller endpoints, but for starters, being able to get slave information and tasks information per framework would be useful. | 5 |
| 214 | MESOS-2166 | PerfEventIsolatorTest.ROOT_CGROUPS_Sample requires 'perf' to be installed | The perf::valid() relies on the 'perf' command being installed. This isn't always the case. Configure should probably check for the perf command exists. | 1 |
| 215 | MESOS-2176 | Hierarchical allocator inconsistently accounts for reserved resources. | Looking through the allocator code for MESOS-2099, I see an issue with respect to accounting reserved resources in the sorters: Within {{HierarchicalAllocatorProcess::allocate}}, only unreserved resources are accounted for in the sorters, whereas everywhere else (add/remove framework, add/remove slave) we account for both reserved and unreserved. From git blame, it looks like this issue was introduced over a long course of refactoring and fixes to the allocator. My guess is that this was never caught due to the lack of unit-testability of the allocator (unnecessarily requires a master PID to use an allocator). From my understanding, the two levels of the hierarchical sorter should have the following semantics: # Level 1 sorts across roles. Only unreserved resources are shared across roles, and therefore the "role sorter" for level 1 should only account for the unreserved resource pool. # Level 2 sorts across frameworks, within a role. Both unreserved and reserved resources are shared across frameworks within a role, and therefore the "framework sorters" for level 2 should each account for the reserved resource pool for the role, as well as the unreserved resources _allocated_ inside the role. | 5 |
| 216 | MESOS-2182 | Performance issue in libprocess SocketManager. | Noticed an issue in production under which the master is slow to respond after failover for ~15 minutes. After looking at some perf data, the top offender is: {noformat} 12.02% mesos-master libmesos-0.21.0-rc3.so [.] std::_Rb_tree<process::ProcessBase*, process::ProcessBase*, std::_Identity<process::ProcessBase*>, std::less<process::ProcessBase*>, std::allocator<process::ProcessBase*> >::erase(process::ProcessBase* const&) ... 3.29% mesos-master libmesos-0.21.0-rc3.so [.] process::SocketManager::exited(process::ProcessBase*) {noformat} It appears that in the SocketManager, whenever an internal Process exits, we loop over all the links unnecessarily: {code} void SocketManager::exited(ProcessBase* process) { // An exited event is enough to cause the process to get deleted // (e.g., by the garbage collector), which means we can't // dereference process (or even use the address) after we enqueue at // least one exited event. Thus, we save the process pid. const UPID pid = process->pid; // Likewise, we need to save the current time of the process so we // can update the clocks of linked processes as appropriate. const Time time = Clock::now(process); synchronized (this) { // Iterate through the links, removing any links the process might // have had and creating exited events for any linked processes. foreachpair (const UPID& linkee, set<ProcessBase*>& processes, links) { processes.erase(process); if (linkee == pid) { foreach (ProcessBase* linker, processes) { CHECK(linker != process) << "Process linked with itself"; synchronized (timeouts) { if (Clock::paused()) { Clock::update(linker, time); } } linker->enqueue(new ExitedEvent(linkee)); } } } links.erase(pid); } } {code} On clusters with 10,000s of slaves, this means we hold the socket manager lock for a very expensive loop erasing nothing from a set! This is because, the master contains links from the Master Process to each slave. However, when a random ephemeral Process terminates, we don't need to loop over each slave link. While we hold this lock, the following calls will block: {code} class SocketManager { public: Socket accepted(int s); void link(ProcessBase* process, const UPID& to); PID<HttpProxy> proxy(const Socket& socket); void send(Encoder* encoder, bool persist); void send(const Response& response, const Request& request, const Socket& socket); void send(Message* message); Encoder* next(int s); void close(int s); void exited(const Node& node); void exited(ProcessBase* process); ... {code} As a result, the slave observers and the master can block calling send()! Short term, we will try to fix this issue by removing the unnecessary looping. Longer term, it would be nice to avoid all this locking when sending on independent sockets. | 3 |
| 217 | MESOS-2184 | deprecate unused flag 'cgroups_subsystems' | cgroups_subsystems is a slave flag that is no longer used and should be deprecated. | 1 |
| 218 | MESOS-2191 | Add ContainerId to the TaskStatus message | {{TaskStatus}} provides the frameworks with certain information ({{executorId}}, {{slaveId}}, etc.) which is useful when collecting statistics about cluster performance; however, it is difficult to associate tasks to the container it is executed since this information stays always within mesos itself. Therefore it would be good to provide the framework scheduler with this information, adding a new field in the {{TaskStatus}} message. See comments for a use case. | 3 |
| 219 | MESOS-2199 | Failing test: SlaveTest.ROOT_RunTaskWithCommandInfoWithUser | Appears that running the executor as {{nobody}} is not supported. [~nnielsen] can you take a look? Executor log: {noformat} [root@hostname build]# cat /tmp/SlaveTest_ROOT_RunTaskWithCommandInfoWithUser_cxF1dY/slaves/20141219-005206-2081170186-60487-11862-S0/frameworks/20141219-005206-2081170186-60 487-11862-0000/executors/1/runs/latest/std* sh: /home/idownes/workspace/mesos/build/src/mesos-executor: Permission denied {noformat} Test output: {noformat} [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from SlaveTest [ RUN ] SlaveTest.ROOT_RunTaskWithCommandInfoWithUser ../../src/tests/slave_tests.cpp:680: Failure Value of: statusRunning.get().state() Actual: TASK_FAILED Expected: TASK_RUNNING ../../src/tests/slave_tests.cpp:682: Failure Failed to wait 10secs for statusFinished ../../src/tests/slave_tests.cpp:673: Failure Actual function call count doesn't match EXPECT_CALL(sched, statusUpdate(&driver, _))... Expected: to be called twice Actual: called once - unsatisfied and active [ FAILED ] SlaveTest.ROOT_RunTaskWithCommandInfoWithUser (10641 ms) [----------] 1 test from SlaveTest (10641 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (10658 ms total) {noformat} | 2 |
| 220 | MESOS-2200 | bogus docker images result in bad error message to scheduler | When a scheduler specifies a bogus image in ContainerInfo mesos doesn't tell the scheduler that the docker pull failed or why. This error is logged in the mesos-slave log, but it isn't given to the scheduler (as far as I can tell): {noformat} E1218 23:50:55.406230 8123 slave.cpp:2730] Container '8f70784c-3e40-4072-9ca2-9daed23f15ff' for executor 'thermos-1418946354013-xxx-xxx-curl-0-f500cc41-dd0a-4338-8cbc-d631cb588bb1' of framework '20140522-213145-1749004561-5050-29512-0000' failed to start: Failed to 'docker pull docker-registry.example.com/doesntexist/hello1.1:latest': exit status = exited with status 1 stderr = 2014/12/18 23:50:55 Error: image doesntexist/hello1.1 not found {noformat} If the docker image is not in the registry, the scheduler should give the user an error message. If docker pull failed because of networking issues, it should be retried. Mesos should give the scheduler enough information to be able to make that decision. | 2 |
| 221 | MESOS-2201 | ReplicaTest.Restore fails with leveldb greater than v1.7. | I wanted to configure Mesos with system provided leveldb libraries when I ran into this issue. Apparently, if one does {{../configure --with-leveldb=/path/to/leveldb}}, compilation succeeds, however the "ReplicaTest_Restore" test fails with the following back trace: {code} [ RUN ] ReplicaTest.Restore Using temporary directory '/tmp/ReplicaTest_Restore_IZbbRR' I1222 14:16:49.517500 2927 leveldb.cpp:176] Opened db in 10.758917ms I1222 14:16:49.526495 2927 leveldb.cpp:183] Compacted db in 8.931146ms I1222 14:16:49.526523 2927 leveldb.cpp:198] Created db iterator in 5787ns I1222 14:16:49.526531 2927 leveldb.cpp:204] Seeked to beginning of db in 511ns I1222 14:16:49.526535 2927 leveldb.cpp:273] Iterated through 0 keys in the db in 197ns I1222 14:16:49.526623 2927 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I1222 14:16:49.530972 2945 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 3.084458ms I1222 14:16:49.531008 2945 replica.cpp:320] Persisted replica status to VOTING I1222 14:16:49.541263 2927 leveldb.cpp:176] Opened db in 9.980586ms I1222 14:16:49.551636 2927 leveldb.cpp:183] Compacted db in 10.348096ms I1222 14:16:49.551683 2927 leveldb.cpp:198] Created db iterator in 3405ns I1222 14:16:49.551693 2927 leveldb.cpp:204] Seeked to beginning of db in 3559ns I1222 14:16:49.551728 2927 leveldb.cpp:273] Iterated through 1 keys in the db in 29722ns I1222 14:16:49.551751 2927 replica.cpp:741] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I1222 14:16:49.551996 2947 replica.cpp:474] Replica received implicit promise request with proposal 1 I1222 14:16:49.560921 2947 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 8.899591ms I1222 14:16:49.560940 2947 replica.cpp:342] Persisted promised to 1 I1222 14:16:49.561338 2943 replica.cpp:508] Replica received write request for position 1 I1222 14:16:49.568677 2943 leveldb.cpp:343] Persisting action (27 bytes) to leveldb took 7.287155ms I1222 14:16:49.568692 2943 replica.cpp:676] Persisted action at 1 I1222 14:16:49.569042 2942 leveldb.cpp:438] Reading position from leveldb took 26339ns F1222 14:16:49.569411 2927 replica.cpp:721] CHECK_SOME(state): IO error: lock /tmp/ReplicaTest_Restore_IZbbRR/.log/LOCK: already held by process Failed to recover the log *** Check failure stack trace: *** @ 0x7f7f6c53e688 google::LogMessage::Fail() @ 0x7f7f6c53e5e7 google::LogMessage::SendToLog() @ 0x7f7f6c53dff8 google::LogMessage::Flush() @ 0x7f7f6c540d2c google::LogMessageFatal::~LogMessageFatal() @ 0x90a520 _CheckFatal::~_CheckFatal() @ 0x7f7f6c400f4d mesos::internal::log::ReplicaProcess::restore() @ 0x7f7f6c3fd763 mesos::internal::log::ReplicaProcess::ReplicaProcess() @ 0x7f7f6c401271 mesos::internal::log::Replica::Replica() @ 0xcd7ca3 ReplicaTest_Restore_Test::TestBody() @ 0x10934b2 testing::internal::HandleSehExceptionsInMethodIfSupported<>() @ 0x108e584 testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x10768fd testing::Test::Run() @ 0x1077020 testing::TestInfo::Run() @ 0x10775a8 testing::TestCase::Run() @ 0x107c324 testing::internal::UnitTestImpl::RunAllTests() @ 0x1094348 testing::internal::HandleSehExceptionsInMethodIfSupported<>() @ 0x108f2b7 testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x107b1d4 testing::UnitTest::Run() @ 0xd344a9 main @ 0x7f7f66fdfb45 __libc_start_main @ 0x8f3549 (unknown) @ (nil) (unknown) [2] 2927 abort (core dumped) GLOG_logtostderr=1 GTEST_v=10 ./bin/mesos-tests.sh --verbose {code} The bundled version of leveldb is v1.4. I tested version 1.5 and that seems to work. However, v1.6 had some build issues and us unusable with Mesos. The next version v1.7, allows Mesos to compile fine but results in the above error. | 3 |
| 222 | MESOS-2205 | Add user documentation for reservations | Add a user guide for reservations which describes basic usage of them, how ACLs are used to specify who can unreserve whose resources, and few advanced usage cases. | 2 |
| 223 | MESOS-2210 | Disallow special characters in role. | As we introduce persistent volumes in MESOS-1524, we will use roles as directory names on the slave (https://reviews.apache.org/r/28562/). As a result, the master should disallow special characters (like space and slash) in role. | 2 |
| 224 | MESOS-2215 | The Docker containerizer attempts to recover any task when checkpointing is enabled, not just docker tasks. | Once the slave restarts and recovers the task, I see this error in the log for all tasks that were recovered every second or so. Note, these were NOT docker tasks: W0113 16:01:00.790323 773142 monitor.cpp:213] Failed to get resource usage for container 7b729b89-dc7e-4d08-af97-8cd1af560a21 for executor thermos-1421085237813-slipstream-prod-agent-3-8f769514-1835-4151-90d0-3f55dcc940dd of framework 20150109-161713-715350282-5050-290797-0000: Failed to 'docker inspect mesos-7b729b89-dc7e-4d08-af97-8cd1af560a21': exit status = exited with status 1 stderr = Error: No such image or container: mesos-7b729b89-dc7e-4d08-af97-8cd1af560a21 However the tasks themselves are still healthy and running. The slave was launched with --containerizers=mesos,docker ----- More info: it looks like the docker containerizer is a little too ambitious about recovering containers, again this was not a docker task: I0113 15:59:59.476145 773142 docker.cpp:814] Recovering container '7b729b89-dc7e-4d08-af97-8cd1af560a21' for executor 'thermos-1421085237813-slipstream-prod-agent-3-8f769514-1835-4151-90d0-3f55dcc940dd' of framework 20150109-161713-715350282-5050-290797-0000 Looking into the source, it looks like the problem is that the ComposingContainerize runs recover in parallel, but neither the docker containerizer nor mesos containerizer check if they should recover the task or not (because they were the ones that launched it). Perhaps this needs to be written into the checkpoint somewhere? | 8 |
| 225 | MESOS-2222 | Add ACLs for the maintenance HTTP endpoints. | In order to authorize the HTTP endpoints for maintenance (to be added in MESOS-2067), we will need to add an ACL definition for performing maintenance operations. | 3 |
| 226 | MESOS-2225 | FaultToleranceTest.ReregisterFrameworkExitedExecutor is flaky | Observed this on internal CI. {code} [ RUN ] FaultToleranceTest.ReregisterFrameworkExitedExecutor Using temporary directory '/tmp/FaultToleranceTest_ReregisterFrameworkExitedExecutor_yNprKi' I0114 18:50:51.461186 4720 leveldb.cpp:176] Opened db in 4.866948ms I0114 18:50:51.462057 4720 leveldb.cpp:183] Compacted db in 472256ns I0114 18:50:51.462514 4720 leveldb.cpp:198] Created db iterator in 42905ns I0114 18:50:51.462784 4720 leveldb.cpp:204] Seeked to beginning of db in 21630ns I0114 18:50:51.463068 4720 leveldb.cpp:273] Iterated through 0 keys in the db in 19967ns I0114 18:50:51.463485 4720 replica.cpp:744] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0114 18:50:51.464555 4737 recover.cpp:449] Starting replica recovery I0114 18:50:51.465188 4737 recover.cpp:475] Replica is in EMPTY status I0114 18:50:51.467324 4741 replica.cpp:641] Replica in EMPTY status received a broadcasted recover request I0114 18:50:51.470118 4736 recover.cpp:195] Received a recover response from a replica in EMPTY status I0114 18:50:51.475424 4739 recover.cpp:566] Updating replica status to STARTING I0114 18:50:51.476553 4739 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 107545ns I0114 18:50:51.476862 4739 replica.cpp:323] Persisted replica status to STARTING I0114 18:50:51.477309 4739 recover.cpp:475] Replica is in STARTING status I0114 18:50:51.479109 4734 replica.cpp:641] Replica in STARTING status received a broadcasted recover request I0114 18:50:51.481274 4738 recover.cpp:195] Received a recover response from a replica in STARTING status I0114 18:50:51.482324 4738 recover.cpp:566] Updating replica status to VOTING I0114 18:50:51.482913 4738 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 66011ns I0114 18:50:51.483186 4738 replica.cpp:323] Persisted replica status to VOTING I0114 18:50:51.483608 4738 recover.cpp:580] Successfully joined the Paxos group I0114 18:50:51.484031 4738 recover.cpp:464] Recover process terminated I0114 18:50:51.554949 4734 master.cpp:262] Master 20150114-185051-2272962752-57018-4720 (fedora-19) started on 192.168.122.135:57018 I0114 18:50:51.555785 4734 master.cpp:308] Master only allowing authenticated frameworks to register I0114 18:50:51.556046 4734 master.cpp:313] Master only allowing authenticated slaves to register I0114 18:50:51.556426 4734 credentials.hpp:36] Loading credentials for authentication from '/tmp/FaultToleranceTest_ReregisterFrameworkExitedExecutor_yNprKi/credentials' I0114 18:50:51.557003 4734 master.cpp:357] Authorization enabled I0114 18:50:51.558007 4737 hierarchical_allocator_process.hpp:285] Initialized hierarchical allocator process I0114 18:50:51.558521 4741 whitelist_watcher.cpp:65] No whitelist given I0114 18:50:51.562185 4734 master.cpp:1219] The newly elected leader is master@192.168.122.135:57018 with id 20150114-185051-2272962752-57018-4720 I0114 18:50:51.562680 4734 master.cpp:1232] Elected as the leading master! I0114 18:50:51.562950 4734 master.cpp:1050] Recovering from registrar I0114 18:50:51.564506 4736 registrar.cpp:313] Recovering registrar I0114 18:50:51.566162 4737 log.cpp:660] Attempting to start the writer I0114 18:50:51.568691 4741 replica.cpp:477] Replica received implicit promise request with proposal 1 I0114 18:50:51.569154 4741 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 106885ns I0114 18:50:51.569504 4741 replica.cpp:345] Persisted promised to 1 I0114 18:50:51.573277 4740 coordinator.cpp:230] Coordinator attemping to fill missing position I0114 18:50:51.575623 4739 replica.cpp:378] Replica received explicit promise request for position 0 with proposal 2 I0114 18:50:51.576133 4739 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 86360ns I0114 18:50:51.576449 4739 replica.cpp:679] Persisted action at 0 I0114 18:50:51.586966 4736 replica.cpp:511] Replica received write request for position 0 I0114 18:50:51.587666 4736 leveldb.cpp:438] Reading position from leveldb took 60621ns I0114 18:50:51.588043 4736 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 81094ns I0114 18:50:51.588374 4736 replica.cpp:679] Persisted action at 0 I0114 18:50:51.589418 4736 replica.cpp:658] Replica received learned notice for position 0 I0114 18:50:51.590428 4736 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 106648ns I0114 18:50:51.590840 4736 replica.cpp:679] Persisted action at 0 I0114 18:50:51.591104 4736 replica.cpp:664] Replica learned NOP action at position 0 I0114 18:50:51.592260 4734 log.cpp:676] Writer started with ending position 0 I0114 18:50:51.594172 4739 leveldb.cpp:438] Reading position from leveldb took 52163ns I0114 18:50:51.600744 4736 registrar.cpp:346] Successfully fetched the registry (0B) in 35968us I0114 18:50:51.601646 4736 registrar.cpp:445] Applied 1 operations in 184502ns; attempting to update the 'registry' I0114 18:50:51.604329 4737 log.cpp:684] Attempting to append 130 bytes to the log I0114 18:50:51.604966 4737 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0114 18:50:51.606449 4737 replica.cpp:511] Replica received write request for position 1 I0114 18:50:51.606937 4737 leveldb.cpp:343] Persisting action (149 bytes) to leveldb took 84877ns I0114 18:50:51.607199 4737 replica.cpp:679] Persisted action at 1 I0114 18:50:51.611934 4741 replica.cpp:658] Replica received learned notice for position 1 I0114 18:50:51.612423 4741 leveldb.cpp:343] Persisting action (151 bytes) to leveldb took 113059ns I0114 18:50:51.612794 4741 replica.cpp:679] Persisted action at 1 I0114 18:50:51.613056 4741 replica.cpp:664] Replica learned APPEND action at position 1 I0114 18:50:51.614598 4741 log.cpp:703] Attempting to truncate the log to 1 I0114 18:50:51.615157 4741 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0114 18:50:51.616458 4737 replica.cpp:511] Replica received write request for position 2 I0114 18:50:51.616902 4737 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 71716ns I0114 18:50:51.617168 4737 replica.cpp:679] Persisted action at 2 I0114 18:50:51.618505 4740 replica.cpp:658] Replica received learned notice for position 2 I0114 18:50:51.619031 4740 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 78481ns I0114 18:50:51.619567 4740 leveldb.cpp:401] Deleting ~1 keys from leveldb took 59638ns I0114 18:50:51.619832 4740 replica.cpp:679] Persisted action at 2 I0114 18:50:51.620101 4740 replica.cpp:664] Replica learned TRUNCATE action at position 2 I0114 18:50:51.621757 4736 registrar.cpp:490] Successfully updated the 'registry' in 19.78496ms I0114 18:50:51.622658 4736 registrar.cpp:376] Successfully recovered registrar I0114 18:50:51.623261 4736 master.cpp:1077] Recovered 0 slaves from the Registry (94B) ; allowing 10mins for slaves to re-register I0114 18:50:51.670349 4739 slave.cpp:173] Slave started on 115)@192.168.122.135:57018 I0114 18:50:51.671133 4739 credentials.hpp:84] Loading credential for authentication from '/tmp/FaultToleranceTest_ReregisterFrameworkExitedExecutor_ONrVug/credential' I0114 18:50:51.671685 4739 slave.cpp:282] Slave using credential for: test-principal I0114 18:50:51.672245 4739 slave.cpp:300] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0114 18:50:51.673360 4739 slave.cpp:329] Slave hostname: fedora-19 I0114 18:50:51.673660 4739 slave.cpp:330] Slave checkpoint: false W0114 18:50:51.674052 4739 slave.cpp:332] Disabling checkpointing is deprecated and the --checkpoint flag will be removed in a future release. Please avoid using this flag I0114 18:50:51.677234 4737 state.cpp:33] Recovering state from '/tmp/FaultToleranceTest_ReregisterFrameworkExitedExecutor_ONrVug/meta' I0114 18:50:51.684973 4739 status_update_manager.cpp:197] Recovering status update manager I0114 18:50:51.687644 4739 slave.cpp:3519] Finished recovery I0114 18:50:51.688698 4737 slave.cpp:613] New master detected at master@192.168.122.135:57018 I0114 18:50:51.688902 4734 status_update_manager.cpp:171] Pausing sending status updates I0114 18:50:51.689482 4737 slave.cpp:676] Authenticating with master master@192.168.122.135:57018 I0114 18:50:51.689910 4737 slave.cpp:681] Using default CRAM-MD5 authenticatee I0114 18:50:51.690577 4741 authenticatee.hpp:138] Creating new client SASL connection I0114 18:50:51.691453 4737 slave.cpp:649] Detecting new master I0114 18:50:51.691864 4741 master.cpp:4130] Authenticating slave(115)@192.168.122.135:57018 I0114 18:50:51.692369 4741 master.cpp:4141] Using default CRAM-MD5 authenticator I0114 18:50:51.693208 4741 authenticator.hpp:170] Creating new server SASL connection I0114 18:50:51.694598 4738 authenticatee.hpp:229] Received SASL authentication mechanisms: CRAM-MD5 I0114 18:50:51.694893 4738 authenticatee.hpp:255] Attempting to authenticate with mechanism 'CRAM-MD5' I0114 18:50:51.695329 4741 authenticator.hpp:276] Received SASL authentication start I0114 18:50:51.695641 4741 authenticator.hpp:398] Authentication requires more steps I0114 18:50:51.696028 4736 authenticatee.hpp:275] Received SASL authentication step I0114 18:50:51.696486 4741 authenticator.hpp:304] Received SASL authentication step I0114 18:50:51.696753 4741 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0114 18:50:51.697041 4741 auxprop.cpp:171] Looking up auxiliary property '*userPassword' I0114 18:50:51.697343 4741 auxprop.cpp:171] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0114 18:50:51.697685 4741 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0114 18:50:51.697998 4741 auxprop.cpp:121] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0114 18:50:51.698251 4741 auxprop.cpp:121] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0114 18:50:51.698580 4741 authenticator.hpp:390] Authentication success I0114 18:50:51.698927 4735 authenticatee.hpp:315] Authentication success I0114 18:50:51.705123 4741 master.cpp:4188] Successfully authenticated principal 'test-principal' at slave(115)@192.168.122.135:57018 I0114 18:50:51.705847 4720 sched.cpp:151] Version: 0.22.0 I0114 18:50:51.707159 4736 sched.cpp:248] New master detected at master@192.168.122.135:57018 I0114 18:50:51.707523 4736 sched.cpp:304] Authenticating with master master@192.168.122.135:57018 I0114 18:50:51.707792 4736 sched.cpp:311] Using default CRAM-MD5 authenticatee I0114 18:50:51.708412 4736 authenticatee.hpp:138] Creating new client SASL connection I0114 18:50:51.709316 4735 slave.cpp:747] Successfully authenticated with master master@192.168.122.135:57018 I0114 18:50:51.709723 4737 master.cpp:4130] Authenticating scheduler-092fbbec-0938-4355-8187-fb92e5174c64@192.168.122.135:57018 I0114 18:50:51.710274 4737 master.cpp:4141] Using default CRAM-MD5 authenticator I0114 18:50:51.710739 4735 slave.cpp:1075] Will retry registration in 17.028024ms if necessary I0114 18:50:51.711304 4737 master.cpp:3276] Registering slave at slave(115)@192.168.122.135:57018 (fedora-19) with id 20150114-185051-2272962752-57018-4720-S0 I0114 18:50:51.711459 4738 authenticator.hpp:170] Creating new server SASL connection I0114 18:50:51.713142 4739 registrar.cpp:445] Applied 1 operations in 100530ns; attempting to update the 'registry' I0114 18:50:51.713465 4738 authenticatee.hpp:229] Received SASL authentication mechanisms: CRAM-MD5 I0114 18:50:51.715435 4738 authenticatee.hpp:255] Attempting to authenticate with mechanism 'CRAM-MD5' I0114 18:50:51.715963 4740 authenticator.hpp:276] Received SASL authentication start I0114 18:50:51.716258 4740 authenticator.hpp:398] Authentication requires more steps I0114 18:50:51.716524 4740 authenticatee.hpp:275] Received SASL authentication step I0114 18:50:51.716784 4740 authenticator.hpp:304] Received SASL authentication step I0114 18:50:51.716979 4740 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0114 18:50:51.717139 4740 auxprop.cpp:171] Looking up auxiliary property '*userPassword' I0114 18:50:51.717315 4740 auxprop.cpp:171] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0114 18:50:51.717542 4740 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0114 18:50:51.717703 4740 auxprop.cpp:121] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0114 18:50:51.717864 4740 auxprop.cpp:121] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0114 18:50:51.718040 4740 authenticator.hpp:390] Authentication success I0114 18:50:51.718292 4740 authenticatee.hpp:315] Authentication success I0114 18:50:51.718454 4738 master.cpp:4188] Successfully authenticated principal 'test-principal' at scheduler-092fbbec-0938-4355-8187-fb92e5174c64@192.168.122.135:57018 I0114 18:50:51.719012 4740 sched.cpp:392] Successfully authenticated with master master@192.168.122.135:57018 I0114 18:50:51.719364 4740 sched.cpp:515] Sending registration request to master@192.168.122.135:57018 I0114 18:50:51.719702 4740 sched.cpp:548] Will retry registration in 746.539282ms if necessary I0114 18:50:51.719902 4735 master.cpp:1417] Received registration request for framework 'default' at scheduler-092fbbec-0938-4355-8187-fb92e5174c64@192.168.122.135:57018 I0114 18:50:51.720232 4735 master.cpp:1298] Authorizing framework principal 'test-principal' to receive offers for role '*' I0114 18:50:51.722206 4735 master.cpp:1481] Registering framework 20150114-185051-2272962752-57018-4720-0000 (default) at scheduler-092fbbec-0938-4355-8187-fb92e5174c64@192.168.122.135:57018 I0114 18:50:51.720927 4737 log.cpp:684] Attempting to append 300 bytes to the log I0114 18:50:51.722924 4737 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I0114 18:50:51.724269 4737 replica.cpp:511] Replica received write request for position 3 I0114 18:50:51.724817 4737 leveldb.cpp:343] Persisting action (319 bytes) to leveldb took 116638ns I0114 18:50:51.728560 4737 replica.cpp:679] Persisted action at 3 I0114 18:50:51.726066 4736 sched.cpp:442] Framework registered with 20150114-185051-2272962752-57018-4720-0000 I0114 18:50:51.728879 4736 sched.cpp:456] Scheduler::registered took 34885ns I0114 18:50:51.725520 4735 hierarchical_allocator_process.hpp:319] Added framework 20150114-185051-2272962752-57018-4720-0000 I0114 18:50:51.731864 4735 hierarchical_allocator_process.hpp:839] No resources available to allocate! I0114 18:50:51.732038 4735 hierarchical_allocator_process.hpp:746] Performed allocation for 0 slaves in 214728ns I0114 18:50:51.733106 4738 replica.cpp:658] Replica received learned notice for position 3 I0114 18:50:51.733340 4738 leveldb.cpp:343] Persisting action (321 bytes) to leveldb took 83165ns I0114 18:50:51.733538 4738 replica.cpp:679] Persisted action at 3 I0114 18:50:51.733705 4738 replica.cpp:664] Replica learned APPEND action at position 3 I0114 18:50:51.735610 4738 registrar.cpp:490] Successfully updated the 'registry' in 21.936128ms I0114 18:50:51.735805 4739 log.cpp:703] Attempting to truncate the log to 3 I0114 18:50:51.736445 4739 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 I0114 18:50:51.737664 4739 replica.cpp:511] Replica received write request for position 4 I0114 18:50:51.738013 4739 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 72906ns I0114 18:50:51.738255 4739 replica.cpp:679] Persisted action at 4 I0114 18:50:51.743397 4734 replica.cpp:658] Replica received learned notice for position 4 I0114 18:50:51.743628 4734 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 78832ns I0114 18:50:51.743837 4734 leveldb.cpp:401] Deleting ~2 keys from leveldb took 63991ns I0114 18:50:51.744004 4734 replica.cpp:679] Persisted action at 4 I0114 18:50:51.744168 4734 replica.cpp:664] Replica learned TRUNCATE action at position 4 I0114 18:50:51.745537 4738 master.cpp:3330] Registered slave 20150114-185051-2272962752-57018-4720-S0 at slave(115)@192.168.122.135:57018 (fedora-19) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0114 18:50:51.745968 4734 hierarchical_allocator_process.hpp:453] Added slave 20150114-185051-2272962752-57018-4720-S0 (fedora-19) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] available) I0114 18:50:51.746070 4735 slave.cpp:781] Registered with master master@192.168.122.135:57018; given slave ID 20150114-185051-2272962752-57018-4720-S0 I0114 18:50:51.751437 4741 status_update_manager.cpp:178] Resuming sending status updates I0114 18:50:51.752428 4740 master.cpp:4072] Sending 1 offers to framework 20150114-185051-2272962752-57018-4720-0000 (default) at scheduler-092fbbec-0938-4355-8187-fb92e5174c64@192.168.122.135:57018 I0114 18:50:51.753764 4740 sched.cpp:605] Scheduler::resourceOffers took 751714ns I0114 18:50:51.754812 4740 master.cpp:2541] Processing reply for offers: [ 20150114-185051-2272962752-57018-4720-O0 ] on slave 20150114-185051-2272962752-57018-4720-S0 at slave(115)@192.168.122.135:57018 (fedora-19) for framework 20150114-185051-2272962752-57018-4720-0000 (default) at scheduler-092fbbec-0938-4355-8187-fb92e5174c64@192.168.122.135:57018 I0114 18:50:51.755040 4740 master.cpp:2647] Authorizing framework principal 'test-principal' to launch task 0 as user 'jenkins' W0114 18:50:51.756431 4741 master.cpp:2124] Executor default for task 0 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases. W0114 18:50:51.756652 4741 master.cpp:2136] Executor default for task 0 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases. I0114 18:50:51.757284 4741 master.hpp:766] Adding task 0 with resources cpus(*):1; mem(*):16 on slave 20150114-185051-2272962752-57018-4720-S0 (fedora-19) I0114 18:50:51.757733 4734 hierarchical_allocator_process.hpp:764] Performed allocation for slave 20150114-185051-2272962752-57018-4720-S0 in 9.535066ms I0114 18:50:51.758117 4735 slave.cpp:2588] Received ping from slave-observer(95)@192.168.122.135:57018 I0114 18:50:51.758630 4741 master.cpp:2897] Launching task 0 of framework 20150114-185051-2272962752-57018-4720-0000 (default) at scheduler-092fbbec-0938-4355-8187-fb92e5174c64@192.168.122.135:57018 with resources cpus(*):1; mem(*):16 on slave 20150114-185051-2272962752-57018-4720-S0 at slave(115)@192.168.122.135:57018 (fedora-19) I0114 18:50:51.759526 4741 hierarchical_allocator_process.hpp:610] Updated allocation of framework 20150114-185051-2272962752-57018-4720-0000 on slave 20150114-185051-2272962752-57018-4720-S0 from cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] to cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0114 18:50:51.759796 4737 slave.cpp:1130] Got assigned task 0 for framework 20150114-185051-2272962752-57018-4720-0000 I0114 18:50:51.761184 4737 slave.cpp:1245] Launching task 0 for framework 20150... | 2 |
| 227 | MESOS-2226 | HookTest.VerifySlaveLaunchExecutorHook is flaky | Observed this on internal CI {code} [ RUN ] HookTest.VerifySlaveLaunchExecutorHook Using temporary directory '/tmp/HookTest_VerifySlaveLaunchExecutorHook_GjBgME' I0114 18:51:34.659353 4720 leveldb.cpp:176] Opened db in 1.255951ms I0114 18:51:34.662112 4720 leveldb.cpp:183] Compacted db in 596090ns I0114 18:51:34.662364 4720 leveldb.cpp:198] Created db iterator in 177877ns I0114 18:51:34.662719 4720 leveldb.cpp:204] Seeked to beginning of db in 19709ns I0114 18:51:34.663010 4720 leveldb.cpp:273] Iterated through 0 keys in the db in 18208ns I0114 18:51:34.663312 4720 replica.cpp:744] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0114 18:51:34.664266 4735 recover.cpp:449] Starting replica recovery I0114 18:51:34.664908 4735 recover.cpp:475] Replica is in EMPTY status I0114 18:51:34.667842 4734 replica.cpp:641] Replica in EMPTY status received a broadcasted recover request I0114 18:51:34.669117 4735 recover.cpp:195] Received a recover response from a replica in EMPTY status I0114 18:51:34.677913 4735 recover.cpp:566] Updating replica status to STARTING I0114 18:51:34.683157 4735 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 137939ns I0114 18:51:34.683507 4735 replica.cpp:323] Persisted replica status to STARTING I0114 18:51:34.684013 4735 recover.cpp:475] Replica is in STARTING status I0114 18:51:34.685554 4738 replica.cpp:641] Replica in STARTING status received a broadcasted recover request I0114 18:51:34.696512 4736 recover.cpp:195] Received a recover response from a replica in STARTING status I0114 18:51:34.700552 4735 recover.cpp:566] Updating replica status to VOTING I0114 18:51:34.701128 4735 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 115624ns I0114 18:51:34.701478 4735 replica.cpp:323] Persisted replica status to VOTING I0114 18:51:34.701817 4735 recover.cpp:580] Successfully joined the Paxos group I0114 18:51:34.702569 4735 recover.cpp:464] Recover process terminated I0114 18:51:34.716439 4736 master.cpp:262] Master 20150114-185134-2272962752-57018-4720 (fedora-19) started on 192.168.122.135:57018 I0114 18:51:34.716913 4736 master.cpp:308] Master only allowing authenticated frameworks to register I0114 18:51:34.717136 4736 master.cpp:313] Master only allowing authenticated slaves to register I0114 18:51:34.717488 4736 credentials.hpp:36] Loading credentials for authentication from '/tmp/HookTest_VerifySlaveLaunchExecutorHook_GjBgME/credentials' I0114 18:51:34.718077 4736 master.cpp:357] Authorization enabled I0114 18:51:34.719238 4738 whitelist_watcher.cpp:65] No whitelist given I0114 18:51:34.719755 4737 hierarchical_allocator_process.hpp:285] Initialized hierarchical allocator process I0114 18:51:34.722584 4736 master.cpp:1219] The newly elected leader is master@192.168.122.135:57018 with id 20150114-185134-2272962752-57018-4720 I0114 18:51:34.722865 4736 master.cpp:1232] Elected as the leading master! I0114 18:51:34.723310 4736 master.cpp:1050] Recovering from registrar I0114 18:51:34.723760 4734 registrar.cpp:313] Recovering registrar I0114 18:51:34.725229 4740 log.cpp:660] Attempting to start the writer I0114 18:51:34.727893 4739 replica.cpp:477] Replica received implicit promise request with proposal 1 I0114 18:51:34.728425 4739 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 114781ns I0114 18:51:34.728662 4739 replica.cpp:345] Persisted promised to 1 I0114 18:51:34.731271 4741 coordinator.cpp:230] Coordinator attemping to fill missing position I0114 18:51:34.733223 4734 replica.cpp:378] Replica received explicit promise request for position 0 with proposal 2 I0114 18:51:34.734076 4734 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 87441ns I0114 18:51:34.734441 4734 replica.cpp:679] Persisted action at 0 I0114 18:51:34.740272 4739 replica.cpp:511] Replica received write request for position 0 I0114 18:51:34.740910 4739 leveldb.cpp:438] Reading position from leveldb took 59846ns I0114 18:51:34.741672 4739 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 189259ns I0114 18:51:34.741919 4739 replica.cpp:679] Persisted action at 0 I0114 18:51:34.743000 4739 replica.cpp:658] Replica received learned notice for position 0 I0114 18:51:34.746844 4739 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 328487ns I0114 18:51:34.747118 4739 replica.cpp:679] Persisted action at 0 I0114 18:51:34.747553 4739 replica.cpp:664] Replica learned NOP action at position 0 I0114 18:51:34.751344 4737 log.cpp:676] Writer started with ending position 0 I0114 18:51:34.753504 4734 leveldb.cpp:438] Reading position from leveldb took 61183ns I0114 18:51:34.762962 4737 registrar.cpp:346] Successfully fetched the registry (0B) in 38.907904ms I0114 18:51:34.763610 4737 registrar.cpp:445] Applied 1 operations in 67206ns; attempting to update the 'registry' I0114 18:51:34.766079 4736 log.cpp:684] Attempting to append 130 bytes to the log I0114 18:51:34.766769 4736 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 1 I0114 18:51:34.768215 4741 replica.cpp:511] Replica received write request for position 1 I0114 18:51:34.768759 4741 leveldb.cpp:343] Persisting action (149 bytes) to leveldb took 87970ns I0114 18:51:34.768995 4741 replica.cpp:679] Persisted action at 1 I0114 18:51:34.770691 4736 replica.cpp:658] Replica received learned notice for position 1 I0114 18:51:34.771273 4736 leveldb.cpp:343] Persisting action (151 bytes) to leveldb took 83590ns I0114 18:51:34.771579 4736 replica.cpp:679] Persisted action at 1 I0114 18:51:34.771917 4736 replica.cpp:664] Replica learned APPEND action at position 1 I0114 18:51:34.773252 4738 log.cpp:703] Attempting to truncate the log to 1 I0114 18:51:34.773756 4735 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 2 I0114 18:51:34.775552 4736 replica.cpp:511] Replica received write request for position 2 I0114 18:51:34.775846 4736 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 71503ns I0114 18:51:34.776695 4736 replica.cpp:679] Persisted action at 2 I0114 18:51:34.785259 4739 replica.cpp:658] Replica received learned notice for position 2 I0114 18:51:34.786252 4737 registrar.cpp:490] Successfully updated the 'registry' in 22.340864ms I0114 18:51:34.787094 4737 registrar.cpp:376] Successfully recovered registrar I0114 18:51:34.787749 4737 master.cpp:1077] Recovered 0 slaves from the Registry (94B) ; allowing 10mins for slaves to re-register I0114 18:51:34.787282 4739 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 707150ns I0114 18:51:34.788692 4739 leveldb.cpp:401] Deleting ~1 keys from leveldb took 60262ns I0114 18:51:34.789048 4739 replica.cpp:679] Persisted action at 2 I0114 18:51:34.789329 4739 replica.cpp:664] Replica learned TRUNCATE action at position 2 I0114 18:51:34.819548 4738 slave.cpp:173] Slave started on 171)@192.168.122.135:57018 I0114 18:51:34.820530 4738 credentials.hpp:84] Loading credential for authentication from '/tmp/HookTest_VerifySlaveLaunchExecutorHook_AYxNqe/credential' I0114 18:51:34.820952 4738 slave.cpp:282] Slave using credential for: test-principal I0114 18:51:34.821516 4738 slave.cpp:300] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0114 18:51:34.822217 4738 slave.cpp:329] Slave hostname: fedora-19 I0114 18:51:34.822502 4738 slave.cpp:330] Slave checkpoint: false W0114 18:51:34.822857 4738 slave.cpp:332] Disabling checkpointing is deprecated and the --checkpoint flag will be removed in a future release. Please avoid using this flag I0114 18:51:34.824998 4737 state.cpp:33] Recovering state from '/tmp/HookTest_VerifySlaveLaunchExecutorHook_AYxNqe/meta' I0114 18:51:34.834015 4738 status_update_manager.cpp:197] Recovering status update manager I0114 18:51:34.834810 4738 slave.cpp:3519] Finished recovery I0114 18:51:34.835906 4734 status_update_manager.cpp:171] Pausing sending status updates I0114 18:51:34.836423 4738 slave.cpp:613] New master detected at master@192.168.122.135:57018 I0114 18:51:34.836908 4738 slave.cpp:676] Authenticating with master master@192.168.122.135:57018 I0114 18:51:34.837190 4738 slave.cpp:681] Using default CRAM-MD5 authenticatee I0114 18:51:34.837820 4737 authenticatee.hpp:138] Creating new client SASL connection I0114 18:51:34.838784 4738 slave.cpp:649] Detecting new master I0114 18:51:34.839306 4740 master.cpp:4130] Authenticating slave(171)@192.168.122.135:57018 I0114 18:51:34.839957 4740 master.cpp:4141] Using default CRAM-MD5 authenticator I0114 18:51:34.841236 4740 authenticator.hpp:170] Creating new server SASL connection I0114 18:51:34.842681 4741 authenticatee.hpp:229] Received SASL authentication mechanisms: CRAM-MD5 I0114 18:51:34.843118 4741 authenticatee.hpp:255] Attempting to authenticate with mechanism 'CRAM-MD5' I0114 18:51:34.843581 4740 authenticator.hpp:276] Received SASL authentication start I0114 18:51:34.843962 4740 authenticator.hpp:398] Authentication requires more steps I0114 18:51:34.844357 4740 authenticatee.hpp:275] Received SASL authentication step I0114 18:51:34.844780 4740 authenticator.hpp:304] Received SASL authentication step I0114 18:51:34.845113 4740 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0114 18:51:34.845507 4740 auxprop.cpp:171] Looking up auxiliary property '*userPassword' I0114 18:51:34.845835 4740 auxprop.cpp:171] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0114 18:51:34.846238 4740 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0114 18:51:34.846542 4740 auxprop.cpp:121] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0114 18:51:34.846806 4740 auxprop.cpp:121] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0114 18:51:34.847110 4740 authenticator.hpp:390] Authentication success I0114 18:51:34.847808 4734 authenticatee.hpp:315] Authentication success I0114 18:51:34.851029 4734 slave.cpp:747] Successfully authenticated with master master@192.168.122.135:57018 I0114 18:51:34.851608 4737 master.cpp:4188] Successfully authenticated principal 'test-principal' at slave(171)@192.168.122.135:57018 I0114 18:51:34.854962 4720 sched.cpp:151] Version: 0.22.0 I0114 18:51:34.856674 4734 slave.cpp:1075] Will retry registration in 3.085482ms if necessary I0114 18:51:34.857434 4739 sched.cpp:248] New master detected at master@192.168.122.135:57018 I0114 18:51:34.861433 4739 sched.cpp:304] Authenticating with master master@192.168.122.135:57018 I0114 18:51:34.861693 4739 sched.cpp:311] Using default CRAM-MD5 authenticatee I0114 18:51:34.857795 4737 master.cpp:3276] Registering slave at slave(171)@192.168.122.135:57018 (fedora-19) with id 20150114-185134-2272962752-57018-4720-S0 I0114 18:51:34.862951 4737 authenticatee.hpp:138] Creating new client SASL connection I0114 18:51:34.863919 4735 registrar.cpp:445] Applied 1 operations in 120272ns; attempting to update the 'registry' I0114 18:51:34.864645 4738 master.cpp:4130] Authenticating scheduler-c45273e4-6eb5-44ee-bf45-71b353db648f@192.168.122.135:57018 I0114 18:51:34.865033 4738 master.cpp:4141] Using default CRAM-MD5 authenticator I0114 18:51:34.866904 4738 authenticator.hpp:170] Creating new server SASL connection I0114 18:51:34.868840 4737 authenticatee.hpp:229] Received SASL authentication mechanisms: CRAM-MD5 I0114 18:51:34.869125 4737 authenticatee.hpp:255] Attempting to authenticate with mechanism 'CRAM-MD5' I0114 18:51:34.869523 4737 authenticator.hpp:276] Received SASL authentication start I0114 18:51:34.869835 4737 authenticator.hpp:398] Authentication requires more steps I0114 18:51:34.870213 4737 authenticatee.hpp:275] Received SASL authentication step I0114 18:51:34.870622 4737 authenticator.hpp:304] Received SASL authentication step I0114 18:51:34.870946 4737 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0114 18:51:34.871219 4737 auxprop.cpp:171] Looking up auxiliary property '*userPassword' I0114 18:51:34.871554 4737 auxprop.cpp:171] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0114 18:51:34.871968 4737 auxprop.cpp:99] Request to lookup properties for user: 'test-principal' realm: 'fedora-19' server FQDN: 'fedora-19' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0114 18:51:34.872297 4737 auxprop.cpp:121] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0114 18:51:34.872655 4737 auxprop.cpp:121] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0114 18:51:34.873024 4737 authenticator.hpp:390] Authentication success I0114 18:51:34.873428 4737 authenticatee.hpp:315] Authentication success I0114 18:51:34.873632 4739 master.cpp:4188] Successfully authenticated principal 'test-principal' at scheduler-c45273e4-6eb5-44ee-bf45-71b353db648f@192.168.122.135:57018 I0114 18:51:34.875006 4740 sched.cpp:392] Successfully authenticated with master master@192.168.122.135:57018 I0114 18:51:34.875319 4740 sched.cpp:515] Sending registration request to master@192.168.122.135:57018 I0114 18:51:34.876200 4740 sched.cpp:548] Will retry registration in 1.952991346secs if necessary I0114 18:51:34.876729 4738 master.cpp:1417] Received registration request for framework 'default' at scheduler-c45273e4-6eb5-44ee-bf45-71b353db648f@192.168.122.135:57018 I0114 18:51:34.877040 4738 master.cpp:1298] Authorizing framework principal 'test-principal' to receive offers for role '*' I0114 18:51:34.878059 4738 master.cpp:1481] Registering framework 20150114-185134-2272962752-57018-4720-0000 (default) at scheduler-c45273e4-6eb5-44ee-bf45-71b353db648f@192.168.122.135:57018 I0114 18:51:34.878473 4739 log.cpp:684] Attempting to append 300 bytes to the log I0114 18:51:34.879464 4737 coordinator.cpp:340] Coordinator attempting to write APPEND action at position 3 I0114 18:51:34.880116 4734 hierarchical_allocator_process.hpp:319] Added framework 20150114-185134-2272962752-57018-4720-0000 I0114 18:51:34.880470 4734 hierarchical_allocator_process.hpp:839] No resources available to allocate! I0114 18:51:34.882331 4734 hierarchical_allocator_process.hpp:746] Performed allocation for 0 slaves in 1.901284ms I0114 18:51:34.884024 4741 sched.cpp:442] Framework registered with 20150114-185134-2272962752-57018-4720-0000 I0114 18:51:34.884454 4741 sched.cpp:456] Scheduler::registered took 44320ns I0114 18:51:34.881965 4737 replica.cpp:511] Replica received write request for position 3 I0114 18:51:34.885218 4737 leveldb.cpp:343] Persisting action (319 bytes) to leveldb took 134480ns I0114 18:51:34.885716 4737 replica.cpp:679] Persisted action at 3 I0114 18:51:34.886034 4739 slave.cpp:1075] Will retry registration in 22.947772ms if necessary I0114 18:51:34.886291 4740 master.cpp:3264] Ignoring register slave message from slave(171)@192.168.122.135:57018 (fedora-19) as admission is already in progress I0114 18:51:34.894690 4736 replica.cpp:658] Replica received learned notice for position 3 I0114 18:51:34.898638 4736 leveldb.cpp:343] Persisting action (321 bytes) to leveldb took 215501ns I0114 18:51:34.899055 4736 replica.cpp:679] Persisted action at 3 I0114 18:51:34.899416 4736 replica.cpp:664] Replica learned APPEND action at position 3 I0114 18:51:34.911782 4736 registrar.cpp:490] Successfully updated the 'registry' in 46.176768ms I0114 18:51:34.912286 4740 log.cpp:703] Attempting to truncate the log to 3 I0114 18:51:34.913108 4740 coordinator.cpp:340] Coordinator attempting to write TRUNCATE action at position 4 I0114 18:51:34.915027 4736 master.cpp:3330] Registered slave 20150114-185134-2272962752-57018-4720-S0 at slave(171)@192.168.122.135:57018 (fedora-19) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0114 18:51:34.915642 4735 hierarchical_allocator_process.hpp:453] Added slave 20150114-185134-2272962752-57018-4720-S0 (fedora-19) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] available) I0114 18:51:34.917809 4735 hierarchical_allocator_process.hpp:764] Performed allocation for slave 20150114-185134-2272962752-57018-4720-S0 in 514027ns I0114 18:51:34.916689 4738 replica.cpp:511] Replica received write request for position 4 I0114 18:51:34.915784 4741 slave.cpp:781] Registered with master master@192.168.122.135:57018; given slave ID 20150114-185134-2272962752-57018-4720-S0 I0114 18:51:34.919293 4741 slave.cpp:2588] Received ping from slave-observer(156)@192.168.122.135:57018 I0114 18:51:34.919775 4740 status_update_manager.cpp:178] Resuming sending status updates I0114 18:51:34.920374 4736 master.cpp:4072] Sending 1 offers to framework 20150114-185134-2272962752-57018-4720-0000 (default) at scheduler-c45273e4-6eb5-44ee-bf45-71b353db648f@192.168.122.135:57018 I0114 18:51:34.920569 4738 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 1.540136ms I0114 18:51:34.921092 4738 replica.cpp:679] Persisted action at 4 I0114 18:51:34.927111 4735 replica.cpp:658] Replica received learned notice for position 4 I0114 18:51:34.927299 4734 sched.cpp:605] Scheduler::resourceOffers took 1.335524ms I0114 18:51:34.930418 4735 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 1.596377ms I0114 18:51:34.930882 4735 leveldb.cpp:401] Deleting ~2 keys from leveldb took 67578ns I0114 18:51:34.931115 4735 replica.cpp:679] Persisted action at 4 I0114 18:51:34.931529 4735 replica.cpp:664] Replica learned TRUNCATE action at position 4 I0114 18:51:34.930356 4734 master.cpp:2541] Processing reply for offers: [ 20150114-185134-2272962752-57018-4720-O0 ] on slave 20150114-185134-2272962752-57018-4720-S0 at slave(171)@192.168.122.135:57018 (fedora-19) for framework 20150114-185134-2272962752-57018-4720-0000 (default) at scheduler-c45273e4-6eb5-44ee-bf45-71b353db648f@192.168.122.135:57018 I0114 18:51:34.932834 4734 master.cpp:2647] Authorizing framework principal 'test-principal' to launch task 1 as user 'jenkins' W0114 18:51:34.934442 4736 master.cpp:2124] Executor default for task 1 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases. W0114 18:51:34.934960 4736 master.cpp:2136] Executor default for task 1 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases. I0114 18:51:34.935878 4736 master.hpp:766] Adding task 1 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 20150114-185134-2272962752-57018-4720-S0 (fedora-19) I0114 18:51:34.939453 4738 hierarchical_allocator_process.hpp:610] Updated allocation of framework 20150114-185134-2272962752-57018-4720-0000 on slave 20150114-185134-2272962752-57018-4720-S0 from cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] to cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0114 18:51:34.939950 4736 master.cpp:2897] Launching task 1 of framework 20150114-185134-2272962752-57018-4720-0000 (default) at scheduler-c45273e4-6eb5-44ee-bf45-71b353db648f@192.168.122.135:57018 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 201... | 3 |
