issuekey,title,description,storypoint DM-4,"Transition git repositories to Stash","Transition gitolite-managed repositories to Atlassian Stash.",10 DM-8,"Finalize DM mission statement","The proposed mission statement for the LSST Software Stack development: {quote} Enabling LSST science by creating a well documented, state-of-the-art, high-performance, scalable, multi-camera, open source, O/IR survey data processing and analysis system {quote} with the following rationale: * well documented -- as otherwise it will be impossible to maintain or be usable for Level 3 * state-of-the-art -- because the quality of the instrument and needs of the science to be systematic limited require us to develop new algorithms an break new ground in a number of areas * high-performance -- because of the massive amount of data we need to process for a reasonable budget * scalable -- because we need it to run from between a single core (for developers, or Level 3 users) to tens of thousands of cores (for Data Release production) * multi-camera -- because we need to process precursor data for development purposes, and because our as-delivered camera won't be ideal. * open source -- because we want the community and future surveys to benefit from our efforts. ",2 DM-9,"Open up LSST software mailing lists","We all benefit from making LSST software development as open as possible and conducive to outside volunteer contributions (*). One way to increase community involvement is to open up our development mailing lists to the public, analogous to the way other open source projects do. For example, we could have: * software-devel@lsstcorp.org: the development mailing list, equivalent to current lsst-data * software-users@lsstcorp.org: the users mailing list, equivalent to current lsst-dm-stack-users mailing list (but it could possibly be replaced by StackOverflow/Confluence Questions) * lsst-dm@lsstcorp.org: internal, DM-staff only mailing list, for the *rare* discussions/notices that should go out to staff only. (*) Though we don't rely on them for meeting the project specs (legally required disclaimer :) ).",1 DM-10,"Transition to Confluence Questions","Open the Confluence Questions site, for interaction with the community (and to enable community self-help). ",10 DM-17,"Add derivatives-based optimizer to meas_multifit","See https://dev.lsstcorp.org/trac/ticket/3146 Story points estimate is for remaining work only (just the code review, which is still substantial).",10 DM-20,"Release EUPS 1.3.0","Release EUPS 1.3.0 in RHL's github repository.",1 DM-21,"Determine final URL/location for W'14 stack","Need to know where W'14 stack files are going to be housed.",1 DM-22,"Update newinstall.sh to check for existence of git and python on users' machines",NULL,1 DM-23,"Confirm stack builds on OS X 10.8",NULL,1 DM-25,"Build Winter'14 release","Run lsst-build scripts and 'eups distrib create' to build the Winter'14 release.",1 DM-26,"Update installation instructions","Update Confluence instructions on how to install the Winter'14 stack. The instructions are at: https://confluence.lsstcorp.org/display/LSWUG/LSST+Software+User+Guide",1 DM-28,"Modify gitolite permissions to allow issue/DM-NNNN branches","Use issue/DM-NNNN branches for issues tracked in JIRA, to differentiate them from tickets/NNNN branches that are still in Trac.",1 DM-52,"Qserv configuration - detailed design","Detailed design covering how all Qserv components will be configured for runtime. ",3 DM-53,"Node configuration and bootstrapping - detailed design","Design covering how all Qserv components will be configured for runtime. Design how new Qserv nodes will be bootstrapped when we add them to the cluster, and how already added nodes will get updated after they were offline (crashed, turned off for maintenance etc) ",6 DM-55,"Node bootstrapping - 1st prototype",NULL,15 DM-56,"Zookeeper-based CSS (v1)",NULL,40 DM-66,"switch back to throwing exceptions in css/Facede.cc",NULL,1 DM-70,"Rewrite xrootd-facing code","This task is related to DMTF-16570-09. It involves plugin in the new XRootD client (which has different interfaces, it is all async etc). We expect to also do a major cleanup on the code that is talking to XRootD while plugging in the new XRootD.",30 DM-71,"Data Distribution Design v1","Need to come up with detailed design covering how we will deal with data distribution: managing multiple replicas, recovering from faults, adding new nodes to the cluster, registering new data from L2 ingest and user data (L3).",8 DM-75,"Modify format of version numbers","The versions auto-generated by the new EUPS+buildbot look like this: {code} ========== $ eups list .... pipe_tasks 7.3.2.0_5_g455c355d0f+070b2c1b35 b61 pyfits 3.1.2+9ef17db9b7 b61 b60 python 0.0.1 b61 b60 scisql 0.3+2b5a2f1b52 b61 b60 scons 2.1.0+2b5a2f1b52 b61 b60 sconsUtils 6.2.0.0_11_gf38997df3e+759c3944a1 b61 sconsUtils 6.2.0.0_19_g755151c0a5+759c3944a1 b60 shapelet 7.3.1.0_1_g9331ee763c+0c72f294dd b61 shapelet 7.3.1.0_1_g9331ee763c+e56ee84a68 b60 skymap 7.3.1.0_1_g64b750c066+db36490146 b60 skymap 7.3.1.0_1_ga6cd540cd3+493a438aa2 b61 skypix 6.1.0.0_1_g1157bf09ae+65137c93cd b61 skypix 6.1.0.0_1_gea33592463+6039c04989 b60 .... ========== {code} where the part before the plus sign is the output of git describe (slightly mangled), and the part after is the SHA1 of the sorted names+sha1s of the dependencies. While this has the benefit that any two causally disconnected buildbots with the same inputs will build the same versions, many people have complained that they're plain ugly. So here's an alternative proposal: * If a tag exist on a commit, use as the version. * If there's no tag, use branchname-gSHA1ABBREV, where any illegal characters in branchname get turned into dots * If the package has dependencies, and a build of this package with different dependencies already exists, append a +N to the end. Keep the mapping of +N -> (dependency name, sha1s) in a special git repository. Given the source code and this git repo, two causally disconnected buildbots will again generate the same set of versions. Example versions: * 7.10.2.1 * 7.10.2.1+5 * master-gdeadbeef * feature.dm-1234-gdeadbeef * feature.dm-1234-gdeadbeef+3 ",1 DM-78,"Save a git-branch when a forced push is detected","Create a gitolite hook that will save a branch when a forced push is detected. E.g., if we have a ticket: 'tickets/DM-AAAA' and someone rebases it and pushes with '--force' before applying the update --- then the hook will branch off the old state into (say): backups/tickets/DM-AAAA/NNNN where NNNN is a monotonically increasing number (per branch).",1 DM-83,"from __future__ import division breaks division of Extent*","If one does: {{from __future__ import division}} the division operator on Extent types raises an exception. How to repeat: I've tried this with v7_3 and master: {code:py} from __future__ import division import lsst.afw.geom as afwGeom npt = afwGeom.Extent2I(10,10)/2 {code} an exception is raised. Removing the first line succeeds as expected.",4 DM-85,"Measurement - Aperture Corrections","The current implementation of aperture corrections in meas_algorithms' CorrectFluxes class is broken, and should not be replicated in meas_base: - it only works when the PSF model is correct - it doesn't work when the aperture to correct to is larger than the PSF model image size - it doesn't propagate the uncertainty in the aperture correction We probably need to do this by estimating the aperture correction and its errors on single frames, using the PSF stars (not the PSF models), then attaching that information to the Psf object to be retrieved *and coadded* by CoaddPsf. JK: In PMCS this would be 10% Bosch J 50% Krughoff S and 40% Owen R Breakdown: jbosch 10%; krughoff 50%; rowen 40%",47 DM-90,"Publish Winter 2014 binaries",NULL,3 DM-92,"tests/testPsfDetermination.py has a broken test","In meas_algorithms tests/testPsfDetermination.py has a test testRejectBlends which does not operate as expected. When it calls pcaPsfDeterminer it results in no usable psf candidates BEFORE blends are rejected. Formerly this resulted in a numpy array named ""sizes"" containing one uninitialized value, which might raise an unexpected exception or raise the desired exception, depending on whether that value was negative or positive. On tickets/DM-3117 I pushed a fix for the bug that caused the invalid ""sizes"" array, but the unit test is now reliably broken because no viable psf candidates raises the wrong exception and does not test blend rejection in any case. So on this same ticket I have commented out the bad test for now.",2 DM-94,"Configure transition screens for DM agile workflow","Whenever an issue is transitioned on JIRA Agile board to 'Ready for Review', a screen should pop up to ask for a reviewer. Whenever it's moved out of that state, another screen should ask for a new assignee.",1 DM-95,"Make lsst-build reuse buildIDs if nothing's changed","All built packages are EUPS-tagged with build IDs (the bNNN EUPS tags). Without this change, new EUPS tags are declared even when nothing changed since the previous build (and EUPS' tags code doesn't scale well at this time). This will be implemented by comparing the newly built manifest against ones stored in versiondb, and reusing the build IDs if a matching one is found.",1 DM-96,"Write unit tests for lsst-build","Unit tests need to be written for lsst-build; they should've been written together with the code, but due to Winter'14 release fire drill they had to be postponed.",10 DM-98,"clean up isr utility code","There is some commented code in isr.py. This should be removed or updated so that it works.",2 DM-148,"Improve naming of getters in AmpInfoTable","The names of the methods to get values from a record on AmpInfoCatalog are potentially confusing. This is because the convention is to call the getters get[attributename]. We could change the method names in the AmpInfoCatalog, or add methods in the SWIG wrapper.",1 DM-177,"Box2I(bbox.getMin(), bbox.getMax()) fails for an empty bbox","Empty Box2I cannot be round tripped: {code:py} from lsst.afw.geom import Box2I b1 = Box2I() b2 = Box2I(b1.getMin(), b1.getMax()) assert b2.isEmpty() {code} It is confusing and surprising that this round tripping fails. It is also a trap for the unwary because saving min and max is the logical way to store boxes in afw tables. Records can contain points but not extents and so it saves casting back and forth and simplifies and clarifies the code to save max instead of extent. Thus that is the path most users will take, and the problem can be a time bomb: it could be quite some time before somebody tries to store an empty box and finds that it does not get retrieved correctly.",1 DM-195,"log4cxx-based logging prototype - v2 ","This is continuation of DMTF-16570-16. Initial work (v1) was done in branch u/bchick/protolog. V2 will include comments sent by K-T and issues discussed at the Qserv meeting March 13 + Free functions vs. a log object need to be discussed more. In particular, when metadata key/value pairs need to be attached, an object might make more sense. Avoiding the getLogger() call when no logging is needed (due to threshold) can be significant. + It's a security breach to use vsprintf() with any user-provided arguments. Use vsnprintf() instead so that you can check for overflow. (Or use stringstream or boost::format.) + In my prototype, I used a combination of a set of static log4cxx::LevelPtr variables with isEnabledFor(level) and a set of cpp macros to avoid the switch. + The return value from getLogger() shouldn't need to be cast. ---- + play with hierarchical names + don't execute code for formatting if debug level is off + use shorter threadId + experiment with defining special python handler, intercept and redirect to our log4cxx-based logging ",15 DM-197,"Replace PositionFunctor with some flavor of XYTransform","afw has a special functor PositionFunctor that acts like an XYTransform. Unless PositionFunctor does not need to be invertible, it makes sense to merge these, likely by replacing PositionFunctor with the transform from afw::image::XYTransformFromWcsPair (as suggested by Jim Bosch on Trac ticket #2214).",1 DM-198,"Rework JOIN support, including Ref*Match tables","Add support to the Ref*Match tables. The relevant code in Qserv core (supporting joins) has already been written. This task is related to DMTF-1640-20",16 DM-199,"Develop new master-worker result system ","Reimplement how results are returned from worker to the czar. Currently it relies on mysqldump, which is fairly inefficient. This is related to DMTF-1650-045",20 DM-201,"Qserv: unit testing (controller module) ","Design and build toy prototype of a test framework for testing controller module. This might require a mock framework, as we want to be able to test things in isolation, without testing everything around the controller module at the same time. This is related to DMTF-16570-20.",10 DM-202,"Qserv: unit testing (query execution) ","Design and build toy prototype of a test framework for testing query execution module. This might require a mock framework, as we want to be able to test things in isolation, without testing everything around the query execution module at the same time. This is related to DMTF-16570-21.",8 DM-203,"Prepare for setting up new cluster at IN2P3 for continuous integration/testing","Once the hardware is available, setup the environment where we could easily run integration testing of different Qserv releases, including testing/comparing performance. Integrate changes implemented in DM-1078 Add install script that exposes individual steps and allows modifications to the config file: newinstall, qserv-configure --prepare, then edit config file (or copy from somewhere), qserv-configure",4 DM-207,"Migrate Qserv czar code to the new logging system","This includes switching Qserv to the new logging. Fine-tuning (what messages are printed deciding on error level) is covered in DM-685.",10 DM-208,"catch exceptions from CSS",NULL,1 DM-210,"S15 Data ingest","Rework existing scripts used to load data into plain mysql. This involves mostly simplifying it, and pushing some functionality like converting types outside of the loading scripts. JK: Refer to loading spreadsheet for PMCS assignments",10 DM-211,"Revise design for Qserv front-end rearchitect ","Revisit the architecture. This includes proxy, down to XRootD client (mysqld, python, zookeeper). Capture all findings in new stories, add these stories to DM-1707",10 DM-212,"Migrate away from using env variables in Qserv","Qserv is currently relying on many env variables. We should migrated away from that to the extend possible.",10 DM-213,"Setup multi-node testbed","It'd be useful to test Qserv using Winter2014 or Summer2014 data set on a multi-node cluster, just to exercise all pieces of the software and double check we are not missing anything.",5 DM-214,"W15 C++ geometry","Port the geometry related code used by Qserv (it is currently written in python) to C++, and switch Qserv to the C++ based version. JK: Refer to loading spreadsheet for PMCS assignments",47 DM-215,"Implement C++ geometry primitives for Qserv",NULL,10 DM-216,"Switch Qserv to C++ geometry primitives",NULL,10 DM-218,"Experiment with no-subchunking based approach",NULL,10 DM-219,"Qserv worker scheduler – code cleanup","The qserv worker scheduler code is a bit ugly. The actual composable scheduler classes (FifoScheduler, ScanScheduler, GroupScheduler, BlendScheduler) might be pretty clean, but the interactions with the rest (wdb, wcontrol) may be harder to understand. There should be a small amount of low-hanging fruit of code to clean up, but to make things more sensible and understandable may require some new abstractions and shuffling of logic to new/different classes. Since this issue was opened, some refactoring work has been done as part of the new xrdssi port, so the organization may be somewhat cleaner now. Still, it's worth it to take a fresh look to evaluate the design/interactions to see how much can/should be reorganized.",6 DM-221,"Implement new (async) XrootD client","(oops, earlier description was meant for DM-878. Sorry for the confusion. -danielw) (I also made a mess with the assignees. I'm sorry. -danielw)",30 DM-222,"Reference Test Server using new XRootD",NULL,5 DM-224,"Switch to MariaDB","We should switch Qserv to the MariaDB Foundation based MySQL.",3 DM-228,"Setup dev test environment","Setup whole Qserv environment, including installing data set, and validate it by running some simple queries. Suggest changes/improvements as appropriate.",8 DM-231,"Refurbish existing configuration ","Refurbish existing configuration scripts to make them work with the new packaging/build system.",10 DM-240,"meas_base plugins for CModel magnitudes","Create meas_base Plugins for single-frame and forced measurement that uses the model-fitting primitives in meas_multifit to implement SDSS-style CModel magnitudes, in which we fit an exp and dev model separately and then fit the linear combination with ellipse parameters held fixed. An old-style plugin has already been implemented on the HSC fork, and should be used as a guide; this issue involves adapting that implementation to meas_base and potentially cleaning it up a bit. Note that the HSC implementation cannot be transferred directly to the LSST side because the meas_algorithms APIs are slightly different on the two forks.",6 DM-241,"refactor forced tasks into two tasks","After looking at it a bit more, I think we should refactor the current meas_base forced photometry task to separate the CmdLineTask from the Measurement task. This will allow the forced measurement task to share a common base class with SingleFrameMeasurementTask (allowing us to move the callPlugin free functions into that base class), and give us better parallels with existing tasks: - ProcesImageForcedTask (my proposed name for the base command-line task) will be more similar to ProcessImageTask. We'll also have ProcessForcedCcdTask and ProcessForcedCoaddTask. - ForcedMeasurementTask will be more similar to SingleFrameMeasurementTask. In short, I think this will both clean up the ugliness in callPlugin and make the whole hierarchy easier for newcomers to understand.",6 DM-242,"switch from '.' to '_' in afw::table fields","We've been mapping '.' to '_' in afw::table I/O, which unnecessarily complicates lots of things. We'd like to switch to using '_' in the field names themselves, which requires ending this mapping in I/O, but we need to be backwards compatible. So we'll add a version to the FITS headers, and continue the mapping if the version is not present or is less than some value. Until we do this, the new field names being used in meas_base won't round-trip.",3 DM-245,"Implement HSC camera in new camera framework","The HSC camera needs to be put in the new camera geometry. I will implement is like we did lsstSim and sdss. That is I will generate a repository with the camera config and ampInfo FITS files that can be unpersisted by the butler. These changes should only require modifying obs_subaru. My plan is to just use the policy file to populate the new geometry, but if there is more up to date information, I'll happily use that.",10 DM-246,"Investigate compensation for Dcr","This is a continuation of the W14 image differencing work. We have characterized the negative effects of Dcr on difference images, and now need to start working on compensation for these effects. This work will also touch on the Wcs and Psf classes, which are probable consumers of this information. Scope includes designing and implementing a class to describe the astrometric effects of Dcr, with consideration as to the other classes (Psf, Wcs) and tasks (ImageDifferencingTask) that may use it",60 DM-247,"Design of Dcr Class","Describes the design process for implementation of a class to model the effects of Dcr. Includes design itself, and the design review process.",20 DM-248,"Implementation of Dcr Class","Core implementation of the class that represents the effects of Dcr. This only includes the initial implementation. We should realistically expect that this class will evolve as it encounters more use cases, no matter how thorough the design process is. ",20 DM-259,"Test and migrate to swig 3.0","-------- Original Message -------- Subject: [LSST-data] Swig 3.0 is out (with C++11 support) Date: Mon, 17 Mar 2014 08:26:05 -0400 From: Robert Lupton the Good To: LSST Data I tried a pre-release on os/x 10.7.5 and it failed some tests, but I haven't tried this version. I had some discussion about this with William, but haven't had time to follow through. R > Date: Sun, 16 Mar 2014 22:44:42 +0000 > From: William S Fulton > > *** ANNOUNCE: SWIG 3.0.0 (16 Mar 2014) *** > > http://www.swig.org > > We're pleased to announce SWIG-3.0.0, the latest SWIG release. > > What is SWIG? > ============= > > SWIG is a software development tool that reads C/C++ header files and > generates the wrapper code needed to make C and C++ code accessible > from other programming languages including Perl, Python, Tcl, Ruby, > PHP, C#, Go, Java, Lua, Scheme (Guile, MzScheme, CHICKEN), D, Ocaml, > Pike, Modula-3, Octave, R, Common Lisp (CLISP, Allegro CL, CFFI, UFFI). > SWIG can also export its parse tree in the form of XML and Lisp > s-expressions. Major applications of SWIG include generation of > scripting language extension modules, rapid prototyping, testing, > and user interface development for large C/C++ systems. > > Availability > ============ > The release is available for download on Sourceforge at > > http://prdownloads.sourceforge.net/swig/swig-3.0.0.tar.gz > > A Windows version is also available at > > http://prdownloads.sourceforge.net/swig/swigwin-3.0.0.zip > > Please report problems with this release to the swig-devel mailing list, > details at http://www.swig.org/mail.html. > > Release Notes > ============= > SWIG-3.0.0 summary: > - This is a major new release focusing primarily on C++ improvements. > - C++11 support added. Please see documentation for details of supported > features: http://www.swig.org/Doc3.0/CPlusPlus11.html > - Nested class support added. This has been taken full advantage of in > Java and C#. Other languages can use the nested classes, but require > further work for a more natural integration into the target language. > We urge folk knowledgeable in the other target languages to step > forward and help with this effort. > - Lua: improved metatables and support for %nspace. > - Go 1.3 support added. > - Python import improvements including relative imports. > - Python 3.3 support completed. > - Perl director support added. > - C# .NET 2 support is now the minimum. Generated using statements are > replaced by fully qualified names. > - Bug fixes and improvements to the following languages: > C#, Go, Guile, Java, Lua, Perl, PHP, Python, Octave, R, Ruby, Tcl > - Various other bug fixes and improvements affecting all languages. > - Note that this release contains some backwards incompatible changes > in some languages. > - Full detailed release notes are in the changes file. ",2 DM-260,"Board workflow modifications","On DM Software Development board, I propose we: * Change ""Ready to Merge"" to ""Review Complete"". Per K-T: {quote} That's what I thought ""Review Complete"" would be -- most of the work is done, but some fixups are needed before merging, and a re-review is not necessary. {quote} * Remove ""Ready for Review"". Instead, the developer should just drag the issue to ""In Review"" and assign it to a reviewer. If/when we re-instate the ""review master"", we may re-introduce ""Ready for Review""",1 DM-271,"Setup the new Buildbot CI system","Setup the Buildbot 0.8.8 testbed for the DM environment. This includes: (1) setting up slaves on the set of common OS on which the DM stack runs; (2) creating a new continuous integration slave using the new eupsPkg-based build and distribution support, Definition of done: * Every git change of master should trigger a build of master * If a build failed, an e-mail will be sent to lsst-data (if the build succeeds, nothing happens) * Failures due to internal buildbot issues (e.g., config problems, transient system availability issues, timeouts, etc.) should go to the buildbot owner. * Allow user-triggered builds via web page (with specified refs to be built), with a common user (until we get LSSTC LDAP directory hooked up). It's understood that locking may not be fully implemented/fleshed out in this story. * It should be possible for a user on lsst-dev to easily setup the stack for either a failed or a successful build. ",100 DM-272,"Move TCT-relevant twiki documentation to Confluence","Congregate all the trac TCT-relevant documents (standards, policies, guidelines, meeting history) onto Confluence.",2 DM-273,"Develop and then create the organizational structure for DM Confluence space","Before we start populating the DM Confluence space with active pages, we should define an overall organizational structure/taxonomy.",1 DM-274,"Adding support for numpy scalar array types as arguments to SWIG wrapped methods","Building against the anaconda on lsst-dev causes construction of Point2I (and other point types) with numpy dtypes to fail with the standard exception: {code} NotImplementedError: Wrong number or type of arguments for overloaded function {code} I did not know that was not allowed. If I am doing it others are as well. I think this should be addressed in some way before W14 release. How to repeat on lsst-dev: {code:sh} setenv EUPS_DIR ~lsstsw/stack/ source ~lsstsw/eups/bin/setups.csh setup afw setup anaconda echo ""import lsst.afw.geom as ag\nimport numpy\nprint ag.Point2I(5,5)\nval = numpy.int64(5)\nprint ag.Point2I(val,val)"" > test.py python test.py {code} Using the old 7_3 stack you can do a similar test: {code:sh} source /lsst/DC3/stacks/default/loadLSST.csh setup -t v7_3 afw echo ""import lsst.afw.geom as ag\nimport numpy\nprint ag.Point2I(5,5)\nval = numpy.int64(5)\nprint ag.Point2I(val,val)"" > test.py python test.py {code}",3 DM-278,"Improve handling errors occuring in AsyncQueryManager","AsyncQueryManager is initialized based on configuration file, if the configuration is invalid, an exception should be thrown (eg in _readConfig()) and gracefully handled upstream.",1 DM-280,"clean up multiple aperture photometry code","I've been doing some minor work on the HSC-side ApertureFlux algorithm, and I wanted to record some concerns here (from both me and RHL) that should be addressed in the new meas_base version: - We should consider merging ApertureFlux and EllipticalApertureFlux into the same algorithm (with a config field to choose whether to use elliptical apertures). We could still register it twice, with a different default config value, and this should eliminate a lot of code duplication. We could also consider having them inherit from a common base class (instead of having EllipticalApertureFlux inherit from ApertureFlux, as is done now). - We should test that the threshold at which we switch from Sinc to naive apertures is obeyed exactly. - We should create a flag for the failure mode in which an aperture cannot be measured because we go off the edge of the image, and test that it appears at the right point. If possible, we should set this flag and measure what area we can within that aperture, instead of just bailing out. - (Somewhat off-topic) We should consider having utility functions on SourceSlotConfig to set all slots to None for use in unit tests.",5 DM-289,"More helpful location information for errors in duplicator/partitioner input","Currently, if the Qserv duplicator or partitioner encounters erroneous input (such as a formatting error, or missing columns), the error message it outputs does not include a mapping back to the location of the error, making it very hard/annoying indeed to fix that erroneous input. Fabrice would (quite reasonably) like to see a filename and line number in the error message. Unfortunately, producing a line number is complicated by the following facts: - multiple threads may be reading sub-sections of the same input file in parallel - sub-sections are not guaranteed to be read in order - lines can have varying length In other words, the code reading/parsing the input has no idea which line it is working on, and making that information available would involve either deferring error reports or extra synchronization (performance loss). What we can easily (and should) do is to arrange for processing code to know the file and byte offset of the input text being processed; error messages should include both pieces of information.",2 DM-290,"Eliminate dependence of query analysis on parser and antlr","I would like to write and compile query analyzer code completely independently of the parser and ANTLR (transitively). This doesn't seem to work right now. This is not currently possible. This might take any where from a day to a week. (I'm not sure if we can finish anything in half a day, if you include the testing, review, feedback, and revision process, but perhaps unit testing will make that faster). Updates to follow after the scope is estimated. Dependencies to be broken: query --> parser, antlr (due to predicate depending on antlr nodes) qana --> parser, antlr ",8 DM-291,"Investigate new XrdSsi interface",NULL,10 DM-295,"xrootd initialization should abort if mysql connection fails","Currently xrootd will happily start even if it can't connect to mysql, it will only print a message: {code} Configration invalid: Unable to connect to MySQL with config: {code} This can be easily overlooked, plus, it is a fatal error and xrootd initialization should be aborted. while working on this, I propose to also improve validateMysql(). At the moment it connects to mysql and database in one call. If connection is fine, but the database does not exist, it will fail without telling user why it failed. This can be confusing. It'd better to connect to mysql without connecting to database, then do ""select_db"", and if that fails, inform the user that the database does not exist. (transferred from trac ticket 3165)",1 DM-296,"fix namespaces in all Qserv core modules","This was suggested by the code review for ticket 1945 (https://dev.lsstcorp.org/trac/wiki/SAT/CodeReviews/1945), pasted below: common/src/*: While it's not required by the coding standards, I'm a big proponent of using namespace scopes in .cc files, which usually save you from needing namespace aliases and will certainly save you from having prefix every declaration with qserv::. At some point I'd recommend changing the header file extension from .hh to .h to match the rest of the LSST DM code, unless it's a big backwards compatibility issue. (transferred from trac ticket 2528)",5 DM-297,"Qserv should check for loaded spatial UDFs","Qserv should have a way of checking for the existence of spatial UDFs loaded in the worker MySQL instances. Obviously, the worker must perform the physical check. However, the worker has no knowledge that spatial UDFs even exist, since the master is responsible for translating spatial spec into UDF call. At the moment, the preferred way of checking would be some sort of administrative command that runs on all nodes. An alternative would be some sanity check that is run on a worker before an admin starts a worker. Or the master could devise a MySQL query to check for things and dispatch to all chunks (although this would not get full coverage when replica exist, while being redundant while workers host more than 1 chunk). (transferred from trac ticket 1959)",2 DM-298,"restarting mysqld breaks qserv","Restaring mysqld results in unusable qserv (even if the restart happens when qserv is completely idle). The error message is: ERROR 2013 (HY000): Lost connection to MySQL server during query This happens most likely because qserv caches the connection, which becomes invalid when server is restarted. I am guessing the same will happen when there is a long period of inactivity (the connection times out). (transferred from trac 2853)",1 DM-300,"Centralize hardcoded constants","Some values in qserv need to become constant(e.g. chunkId column names, dirs, filenames). Some of these are configurable, others are hardcoded in non-obvious places in the code. When multiple places need this value, they really need to agree, and unfortunately, qserv doesn't have a well-known place for these constants yet. Any (constant) value that is needed by different parts of the code needs to be managed in a way that is reasonably obvious to unfamiliar programmers. List of values: * chunkId, subChunkId column names (master...indexing.py, app.py) * environment variable names * + others. This should actually be fairly simple to implement, once the right (?) design is conceived and worked-out. (transferred from trac ticket #2405)",6 DM-309,"Jira for Qserv","Jira setup for Qserv, includes things like adding new tasks, transferring tasks from trac, epic/story/task division, assigning story points, setting scrum board, just learning things and more...",8 DM-312,"Come up with a standard to handle C++ Exceptions in Qserv (and the rest of DM?)","Currently CSS is using one class and relies on different error codes to differentiate between different type of exceptions, while other parts of Qserv core define an exception for each different error. It'd be good to standardize and use the same approach. ",4 DM-313,"cleanup includes in Qserv core modules","Includes need cleanup: group into standard lib, boots and local, sort as appropriate etc. Also, unify forward declarations.",2 DM-318,"CSS - surviving mysql and zookeeper glitches","CSS should gracefully recover from failures such as lost connection to mysqld or zookeeper. It is not clear if it would survive such glitches right now -- this needs to be tested and the code improved as necessary.",4 DM-319,"Create a board (virtual or otherwise) with pictures and names of everyone in DM","I tried compiling a list of everyone who works or has worked on the DM code -- it was nearly impossible. We should have a (public) list both of current staff and our alumni (and where they are now).",1 DM-321,"Re-think thread.cc and dispatcher.cc python interface","The mess of thread.cc and dispatcher.cc need to be re-thought and re-designed so that the interface is smaller and more obvious. ",2 DM-322,"Trim python importing by czar in app.py","Clean up the way modules are imported in qserv master, use relative import when appropriate instead of lsst.qserv.master. (migrated from Trac #2369)",2 DM-326,"Libraries being built in lib64 on OpenSUSE, when EUPS tables assume lib","A report from Darko Jevremovic : {quote} Hi Mario, I managed to build stack v8 on OpenSuse13.1 There were standard problems with lib/lib64 - namely system builds libraries in $PREFIX/lib64 and some programs are hard wired for $PREFIX/lib if you could change the last line of mysqlclient-5.1.65+3/ups/eupspkg.cfg.sh from (cd $PREFIX/lib && ln -s mysql/* . ) to ( cd $PREFIX && if [ ! -f ""lib"" ] ; then ln -sf lib64 lib; fi &&cd $PREFIX/lib && ln -s mysql/* . ) or something along that line (am not sure whether the syntax would work). Also if you could add in the same manner to ups/eupspkg.cfg.sh ( cd $PREFIX && if [ ! -f ""lib"" ] ; then ln -sf lib64 lib; fi) for the following packages: minuit2 gsl cfitsio wcslib {quote} ",1 DM-327,"Take RAM into account when computing NCORES to use in installs","Darko Jevremovic reported he's had to switch off hyperthreading and manually override NCORES, MAKEFLAGS and SCONFLAGS because his 8-core machine had too little RAM to build afw with -j 8. To fix this, eupspkg default build routines should take RAM into account when computing the level of build parallelism. In the meantime, we should document the workaround (contact darko@aob.rs).",1 DM-330,"Local lsst-build invocations should use a different build number prefix","Buildbot-invoked lsst-build installs packages in the stack with ""b#"" tags. These are propagated to the distribution server by {{eups distrib create}}. Local stacks maintained with lsst-build should use different tags so that they don't conflict with these distributed tags.",1 DM-331,"S14 Enhancements in Qserv installation procedure",NULL,15 DM-334,"Cut Qserv release","It'd be very useful to have fully functioning Qserv release with the latest set of changes (build, packaging, CSS, Daniel's fixes etc) during the Hackathon week.",2 DM-335,"Migrate std::lists to std::vectors","Suggested by Andy when reviewing DM-296, discussed at Qserv mtg 3/27. std::list --> std::vector * why? Default now is vector, iterating over vector is much more efficient than over list * revisit on case by case bases, do not blindly replace * preferred solution: typedef, and name it in a way that conveys the intent (e.g., might call it a ""container""), underneath use vector",8 DM-336,"improve code that initializes shared_ptrs ","Reported by Andy when reviewing DM-296. Discussed at Qserv mtg 3/27. boost::shared_ptr =(new T())"" --> boost::make_shared()",4 DM-337,"removed dead code in stringUtil.h","Remove obsolete strToDoubleFunc (and more) in util/stringUtil.h.",1 DM-339,"Make it easy to build and release point releases",NULL,4 DM-342,"Establish github mirror of LSST repositories",NULL,4 DM-343,"Transition to Stash",NULL,10 DM-345,"Enable gravatars","Could you enable gravatars for all our atlassian products (at least Jira + Agile; Confluence) https://confluence.atlassian.com/display/AOD/Configuring+Gravatar+support",1 DM-354,"Add cameraGeom overview to Doxygen documentation","The CameraGeom package needs an overview page (part of afw's main.dox) as part of the Doxygen documentation. I think it's up to Simon or me to add this.",2 DM-355,"Install and tag multiple Qserv versions on the same distserver","Done in DM-366",1 DM-359,"Simplify Co-add example in Software User Guide","The current example in the LSST Software User Guide for co-addition reflects the processes necessary to perform a DR production. While thorough, it only really works on the lsst cluster. The example should be simplified to work on a smaller subset of data, and on single-user machines. Definition of Done: * Following the documentation, it will be possible for users to identify the SDSS data they need (the subset of files) * There will be instructions on how to download the necessary files to their local machine * There will be instructions on how to build the necessary repositories * There will be instructions on how to run the Co-Add+forced photometry tasks. * Any issues requiring access rights to LSST machines or databases will be identified and issues created for later.",8 DM-365,"Integration tests dataset should be packaged in eupspkg","A qserv-integration-tests package should be created : - it would allow to manage easily, in ups/qserv.table, tests version for a given Qserv version. - it would allow to install Qserv dependencies related to testing, like partition (and other data ingest code which may arrive.",3 DM-366,"Refactor install/distribution procedures using lsst-sw"," Here's Andy Salnikov remark, during #3100 review : https://dev.lsstcorp.org/trac/ticket/3100#comment:18",5 DM-367,"CSS performance optimizations (avoiding redundant checks)","Facade.cc: it seems worthwhile to think about how to tweak the implementation to reduce redundant calls. e.g., getChunkedTables() calls _cssI->exists() for the database for each contained table. It also calls exists() for each table, which it just retrieved. In reality, it only needs to call exists() once, for the db. So we are doing 1+3t reads rather than 1+t. (This came up in the review of DM-56, the review comments are captured in DM-225) ",8 DM-368,"integrate qserv_admin backend into czar or separate admind","client/qserv_admin_impl.py: I know the czar's Python code is ugly. But the concept of having more than one entry point for users is also ugly. I feel like the client should just wrap up stuff into JSON and make a REST call into the czar. Ideally the proxy would use REST, perhaps via: https://github.com/fperrad/lua-Spore/, which at least seems more thought-out than lua-xmlrpc. (This came up in the review of DM-56, the review comments are captured in DM-225)",15 DM-369,"Improve how CSS exceptions are handled","CSS has one class for all exceptions. The model adopted by Daniel (each type as a separate exception) is better, as we can catch individual exceptions instead of (a) having to catch all css exceptions, (b) checking type and (c) rethrowing if it is not the type we want. To be able to catch all CSS exceptions, we can just introduce a new base class (that is new comparing to Daniel's version).",6 DM-370,"improved how default values for CSS are handled","Need to improve how defaults are handled in qserv_admin. There seems to be some desire to warn when values are not set--how about setting defaults and just printing what configuration is being used? If this is something human-created, we should have reasonable defaults and not bother the user, unless no default is viable. I think we should only be strict on machine-generated input, where we would like to catch bugs as soon as possible. (This came up in the review of DM-56, the review comments are captured in DM-225)",5 DM-372,"fix testQueryAnalysis","5 tests fail in the testCppParser.",2 DM-373,"Catch AttributeError problems in czar","I wonder if we could catch more exceptions in czar to simplify debugging. For example, if I change: {code} --- a/core/modules/czar/lsst/qserv/master/app.py +++ b/core/modules/czar/lsst/qserv/master/app.py @@ -418,7 +418,7 @@ class InbandQueryAction: self.constraints.size()) dominantDb = getDominantDb(self.sessionId) dbStriping = getDbStriping(self.sessionId) - if (dbStriping.stripes < 1) or (dbStriping.subStripes < 1): + if (dbStriping.x < 1) or (dbStriping.y < 1): {code} It will return to client a very cryptic error: {code} Qserv error: Unexpected error: (, AttributeError('x',), ) {code} with no other clues, traceback or information in the log. ",2 DM-375,"S14 Improve error handling in all parts of Qserv, report sensible errors to users","Qserv needs to handle error gracefully. No matter what error occurs, it should try to automatically recover, and if it can't it should report a reasonable error to user. We should try to poke around and trigger various errors in random places in Qserv and watch what happens, how system fails and/or how it reports the error to user (then we should fix it...). I expect we will need to design a better error handling framework to achieve graceful error handling.",60 DM-380,"loadLSST bug(s) for csh, ksh","A flaw in the v8.0 loadLSST scripts (and/or in eups/bin/setups) causes the following errors: 1) When using ksh: {code} $INSTALL_DIR/loadLSST.ksh ksh: /path/to/INSTALL_DIR/eups/bin/setups.ksh: cannot open [No such file or directory] {code} And indeed, there is no eups/bin/setups.ksh file. 2) When attempting to run the installation demo (v7.2.0.0): {code} $> printenv SHELL /bin/tcsh {code} [The same issue appears with csh, unsurprisingly.] {code} $> source /path/to/install_dir/loadLSST.csh $> cd /path/to/demo $> setup obs_sdss $> ./bin/demo.sh ./bin/demo.sh: line 7: /volumes/d0/lsst/stack80/eups/*default*/bin/setups.sh: No such file or directory ./bin/demo.sh: line 12: setup: command not found {code} After hand-editing the demo.sh script to omit the ""/default"" string from the offending line, the demo runs normally to completion. Note that everything works fine for bash with v8.0, which is what I tested awhile back. ",1 DM-384,"Add Versioning to SourceTable in lsst::afw::table","Add version to afw::table::SourceTable. Persist that version number to fits file when the table is saved, and restore when the table is restored. Tables created and saved to disk prior to this modification will have the version number 0, by default. Tables created with the S14 version will have the version number 1. This change is to enable a new version of slots and field naming conventions as needed by the Measurement Framework overhaul, at the same time allowing current clients of SourceTable to continue to function. The work to define and persist the slots depending on the version will be on a separate issue. Should not appear as an alterable member of the metadata, but should be saved with the metadata and reloaded when the file is reloaded. getVersion and setVersion methods will be used to allow clients to alter this number.",1 DM-386,"Create Command/Event Sender","Simulate the OCS and CCS (via the OCS) sending message to the Base DMCS. Write a library to send commands via method calls, which will be used by commandable entities and by the Base DMCS. The method calls in this library will be used to simulate sending commands from the OCS. This will initially be developed using DM messages, and later switched to use DDS. Write a command line tool to send these messages. Commands with no arguments: init, enable, disable, release, stop, abort, reset. Commands with arguments: configure - arguments are: Set of computers, software and versions to be executed, parameters used to control that software. Events with arguments: startIntegration, nextVisit. Definition of done: * A library that has method calls to each of these commands/events. Each method call sends one message to the given Topic. * Command line tool that can send any of these commands/events to a commandable entity subscribed to a Topic. * Unit tests for each of the commands/events",6 DM-387,"Build Base DMCS communications library","Write a library to be used by each commandable entity and the Base DMCS. Methods in this library receiving commands from the simulated OCS. Specific command actions will be handled by each entity, and events are handled by the Base DMCS. This will initially be developed using DM messages, and later switched to use the DDS. The library will include: * An object with methods for blocking receives, blocking receives with timeouts, and non-blocking receives. Any commands received by these methods are given to another class to call appropriate action methods. * An abstract class implementing each of the following methods for commands: init, configure, enable, disable, release, stop, abort, reset. * An abstract class with methods for implementing the following methods for events: startIntegration and nextVisit",6 DM-388,"Build replicator","Replicator On boot: * Establishes connection with single, pre-configured distributor. * Checks connection with the network outage buffer, the Base raw image cache, and the tape archive. On success, the replicator registers itself with the Orchestration Manager in the fully-operational pool. From K-T: Note that what to check should eventually be dependent on what was specified in the configuration. That way, if the tape library is down, we could still run using the network outage buffer or raw image cache as the redundancy source. In operation: * Receives a job with visit id, exposure sequence number within the visit and raft id. * Queries the Base EFD replica for information needed to process the image. * Subscribes to the Camera Data System (CDS) “startReadout” event and to the OCS telemetry topics. * Waits for a startReadout event * Request the cross-talk corrected exposure for the raft using the CDS client interface and block until it is available. On receipt of crosstalk-corrected image: * Verify integrity of image (hash or checksum) * Send image with telemetry to the distributor, compressing it if the configuration says to do so. At the same time, the image is written to the network outage buffer and the raw image cache using the Data Access Client Framework, retrying until successful for a configured number of tries. All images that are written are tagged with the Archiver mode (by the replicator job). * Request the raw image using the CDS client interface and block until it is available. On receipt of the raw image: * Verify integrity of image (hash or checksum) * Send image to the distributor and simultaneously write to the network outage buffer and the tape archive. ALL image transmission statuses (successful and unsuccessful) are recorded to the Alert Production Database. NOTE: Image some calibration or engineering modes, there may only be raw-image data, not crosstalk-corrected image data. The replicator job configuration will provide for this). NOTE: Replicator jobs will need to detect they are not completing in the expected amount of time. NOTE: Something (the Base DMCS? The Event/Message Monitor?) needs to pay attention to the number of replicator re-registering themselves with the local-only pool and notify NetOps to investigate and resolve the issue. On errors sending or keeping connected to the distributor, the replicator: * unregisters from the “fully-operational” pool * registers with the “local-only” pool The replicator will continue to try and connect to the distributor, and if it is able to reestablish the connection: * unregisters from the “local-only” pool * registers with the “fully-operational” pool Heartbeat messages between replicators and distributors will indicate when the replicators will be able to re-register with the fully-operational pool.",10 DM-389,"Build distributor","Distributor On boot: * Checks the network, archive raw image cache and the tape archive. After all have tested successfully, the distributor waits for a connection from its associated replicator. In operation: * On receipt of a visit id, exposure sequence number and raft id from its associated replicator, the distributor publishes them along with its network address to the Archive DMCS. On receipt of crosstalk-corrected image and associated telemetry from its associated replicator: * Verifies integrity using a hash or checksum * Writes to the raw image cache using the Data Access Client Framework * Decompresses (if necessary) * Separates into individual CCD-sized portions * Sends portions to the appropriate connected Workers Note: Workers can connect to request a CCD-sized crosstalk-corrected image. On receipt of raw image: * Writes it to the tape system. * All images are tagged with the Archive mode. Should include heartbeat monitoring of the replicator/distributor connection.",10 DM-405,"Write Linux Standard Base - compliant init.d scripts","Qserv services init.d scripts have to rely on LSB, in order to work on multiple systems. Remark : xrootd has to be launched as a background process (i.e. with a & at the end). But this always send of return code equal to 0, even if xrootd fails to start, a shell function wait_for_pid will be implemented in xrootd init.d scritps to solve that (inspired from mysqld init.d script).",5 DM-413,"Simulate computation and production of VOEvents for worker batch job","Simulate jobs which do alert processing. Produce VOEvents based on those results.",6 DM-417,"finish adding aliases to afw::table::Schema","This issue picks up the partially-completed Trac Ticket #2351, which adds a string-substitution-based alias mechanism to afw::table::Schema. There are still some issues to sort out w.r.t. constness - in other respects, a Table or Catalog's schema cannot be changed once the Table has been constructed, but we do need to be able to change aliases after table construction.",6 DM-419,"Use aliases in slots",NULL,2 DM-420,"Remove measurement code from meas_algorithms",NULL,4 DM-421,"add basic FunctorKeys","Add the basic FunctorKeys mechanism, and enough implementations to support Slots.",4 DM-422,"Add FunctorKeys to replace compound field functionality","Add more FunctorKeys to replace the functionality in all current compound field types and meas_base result objects. May involve moving some meas_base definitions to afw.",4 DM-423,"Add FunctorKeys for common analysis tasks","Add FunctorKeys for simple, common, calculated fields, including: - Magnitudes from fluxes - Coords from Points, Points from Coords - Ellipse conversions and radius/ellipticity extraction",4 DM-424,"Integrate FunctorKeys with SchemaMapper",NULL,10 DM-425,"Remove support for compound field types",NULL,2 DM-428,"add live DS9-based debugging to measurement framework","The old measurement framework had a lot of live DS9-based display options. We should ensure the new one has at least as many, and that it still works. At some point, we should consider a different mechanism for enabling those displays and possibly other tools for displaying them, but that's out of scope for this issue.",6 DM-429,"Make NoiseReplacer outputs reproduceable","We need a way to get back the noise-replaced Exposure as it was when a particular source was measurement, after the measurement has been run, without having to run noise-replacement on all the previous objects again. There is already code in afw::math::Random to output its state as a string; I think we should probably just save this string in the output catalog. This will require some API changes to allow the NoiseReplacer to modify the schema and set a field in the output records.",3 DM-430,"Control log levels on a per-plugin basis","We should be able to control log levels so that certain plugins are run at one level while the rest are run at another (to allow a particular plugin being debugged to be more verbose).",4 DM-433,"Add slot support for meas_base-style outputs","The slot mechanism in afwTable currently uses compound fields to save the 3 slot types: flux, centroid, and shape. Since the new measurement framework uses a flattened representation in the SourceTable where these types are saved as multiple scalar fields, the slot mechanism need to be altered to handle this new table type. 1. An alternative to KeyTuple for storing the keys required by the slot 2. Fixup get(Centroid, Flux, Shape) in SourceRecord to use correct keys. 3. Fixup the single value getters (getX, getY, etc) to use the correct keys. 4. Persist slot info to fits correctly, based on table version.",4 DM-435,"add aperture-correction measurement code to the end of calibrate","At the end of CalibrateTask, we'll want to compute the PSF and aperture fluxes of the PSF stars, and send those to the PSF model to be stored and interpolated (using the featured added via DM-434). We'll also need to run any other flux measurement algorithms that need to be tied to the PSF fluxes on these same stars; because these can be somewhat slow, we probably want to limit these measurements to only the PSF stars, rather than requiring all these algorithms to be run as part of calibrate.measurement. The relationships between these fluxes and the PSF fluxes will be additional fields to be added to and interpolated by the PSF. The HSC implementation of this work (as well as that of DM-436) was done on issue HSC-191: https://hsc-jira.astro.princeton.edu/jira/browse/HSC-191 There were changes to many packages, but the relevant ones for LSST are: https://github.com/HyperSuprime-Cam/afw/commit/057fb3c0581c512d5664f1883a72da950c9eae9d https://github.com/HyperSuprime-Cam/meas_algorithms/compare/HSC-3.0.0...u/jbosch/DM-191 https://github.com/HyperSuprime-Cam/pipe_tasks/compare/4c3a53e7238cbe9...u/jbosch/DM-191",8 DM-436,"apply aperture corrections in measurement tasks","We need to interpolate the aperture correction to the position of every source, and apply this correction to all appropriate fluxes. The HSC implementation of this work (as well as that of DM-435) was done on issue HSC-191: https://hsc-jira.astro.princeton.edu/jira/browse/HSC-191 There were changes to many packages, but the relevant ones for LSST are: https://github.com/HyperSuprime-Cam/afw/commit/057fb3c0581c512d5664f1883a72da950c9eae9d https://github.com/HyperSuprime-Cam/meas_algorithms/compare/HSC-3.0.0...u/jbosch/DM-191 https://github.com/HyperSuprime-Cam/pipe_tasks/compare/4c3a53e7238cbe9...u/jbosch/DM-191 Note that on the LSST side, we'll want to apply the aperture corrections either within a new plugin in meas_base or as a new part of BaseMeasurementTask, not as a change to the CorrectFluxes algorithm (which will be removed in the future along with the rest of the old measurement framework in meas_algorithms).",7 DM-441,"Setup of four new measurement algorithms for processCcd testing","The goal of this story is to take the following algorithms and make them fully operational with processCcd. PsfFlux,SdssShape,SdssCentroid,SincFlux. This code was moved to meas_base in w14, but has yet to be used in full operation. This is the first step in that process. Each algorithm will at least have one test to confirm that it works, but the unit tests will be very simple. The actual confirmation of full operability will the to run tests against real data, and compare against the existing meas_algorithms. The algorithm code will still require cleanup even after this story is completed. That is because it has intentionally been left the same as what existed in meas_algorithms. The goal of this ticket is to confirm that the each algorithm can (1) complete its measurement with some reasonable result, (2) responds to its major configuration options, and (3) handles at least some of its defined exceptions (that is, flags) correctly. Confirmation that the results are identical with meas_algorithms is a subsequent ticket Cleanup and documentation of this code will be done in a subsequent ticket.",4 DM-443,"Approve/Acquire HipChat licenses","-------- Original Message -------- Subject: Getting HipChat licenses Date: Wed, 09 Apr 2014 08:51:46 -0700 From: Mario Juric Organization: Large Synoptic Survey Telescope Inc. To: Jeffrey Kantor , Iain Goodenow , Stefan Dimmick Jeff et al., I'd like to start the process to acquire HipChat licenses (we have 10 days left on our evaluation one). Right now we have 26 users, so our first total would be $52/month. I'm expecting it will eventually grow to ~50 users or so. In terms of an account to charge, this should really be considered a project-wide tool, but if that's going to cause unnecessary delays I'd propose we charge it to DM as a sign of our infinite kindness and good will :). HipChat wants to simply bill a credit card -- once we get the necessary approvals, Iain, I can add you as an admin, and you can use our corporate one if that's OK. ",1 DM-444,"Write job ads for Tucson DM positions",NULL,4 DM-445,"Test four algorithms for compatibility with original meas_algorithms","Do a trial run of a small area of sky (using a single exposure from measMosaicData). First create a source catalog using the old measurement.py, then run the same test with the measurement task in sfm.py. Compare the results. If the code has been ported correctly, they should match.",4 DM-446,"Setup PeakLikelihoodFlux with new Algorithm Framework","Move PeakLikehoodFlux to meas_base framework ",1 DM-447,"Setup Flux algorithms for testing with processCcd","Similar to DM-441 for flux algorithms GaussianFlux and NaiveFlux Unit tests for major config options, just to be sure that they do something reasonable. Test of at least one exception (flag)",2 DM-448,"Test Centroid algorithms against meas_algorithms",NULL,2 DM-449,"Test Flux algorithms agains meas_algorithms","Test for compatibility of NaiveFlux, GaussianFlux, and PsfFlux against meas_algorithms ",2 DM-454,"reimplement shapelet PSF approximations","The CModel code we want to transfer from HSC in DM-240 currently relies on the old ""multishapelet.psf"" algorithm in meas_extensions_multiShapelet. That means we either need to convert that PSF-to-shapelets code in meas_extensions_multiShapelet to use the meas_base interface, or we need add new PSF-to-shapelets code in meas_multifit. I think the latter is the better choice, even if we delay DM-240 as a result; the heavy algorithmic code is already available as primitives in meas_multifit, so it should just be a matter of packing those into a simple driver, creating a config class for it, and testing it on a few real and simulated PSFs to learn reasonable defaults for the configs.",6 DM-460,"Implement backup/restore for CSS","It'd be very useful to have a way to ""dump"" and ""restore"" entire contents of the key/value store in zookeeper. The dump part is pretty much there (can dump to an ascii file) ",4 DM-461,"Add Classes of MeasurementError","Some measurement failures are global for a whole exposure, such as a missing Psf or Wcs. The framework currently does not distinguish this from a failure in a single measurement. Should add a new subclass of MeasurementError which can be thrown in these cases. Should also add configuration option to the measurement framework to determine what should be done with this type of error. We should also add an exception class and associated how-to-handle config for problems that indicate that something has gone wrong in pre-measurement processing, such as NaNs in the image.",4 DM-463,"PixelsFlags, SkyCoordAlgorithm, and Classification","SkyCoord was moved to DM-441 - done in Python Classification is also simple if done in Python Pixel Keys will be done in C++ Some work left from SdssShape will be done with this ticket Also, since these are out only two Python algorithms in the base set, I will add the exception handling and base fail() methods at this time. ",2 DM-464,"add and use ""suspect"" flag in slots and slot-like measurements","We need to have both ""fatal"" and ""suspect"" flags to handle different levels of warnings in measurement. (other tasks previously included on this ticket have been split into other tickets, including DM-461 and DM-984)",4 DM-466,"lsst-build updates based on feedback from 1st month of use","change lsst-build interface to: * rename 'prepare' to 'clone' * use current directory to clone to * remove the default REPOSITORY_PATTERN, as it isn't complete * use the current username as default build tag prefix * don't write the manifest by default; use --manifest= option instead * refuse to change/overwrite repositories if they're dirty, use --force to override * create a separate 'version' verb * ProductFetcher.fetch doesn't need to return ref, sha1 * read config file within build directory, .btconfig * rename the binary to bt * have the versioner use git db by default, but fall back to generating versions from hash by default See the linked web page for a mock of the command line interface.",6 DM-467,"Understand galfast bugs",NULL,2 DM-468,"Alias measurement.plugins to measurement.algorithms","The config item in the old measurement task, measurement.algorithms was changed to measurement.plugins in meas_base. The creates a backward compatibility issue for code which refers to this class member. Jim's suggested fix is to alias plugins with algorithms in the new measurement task.",1 DM-470,"Rework exceptions in css (python side)","Rework exceptions in css/KwInterface.py: split into key-value related exceptions, possibly moving the rest that deals with db/tables into client. This came up in the CSS review, see DM-225: ""CssException feels a bit out of place....""",1 DM-471,"Create LSSTsim processing example for SW User Guide","Create a worked example in the Software User Guide of processing raw data with obs_lsstSim. The example should include: - Pointers to documents for creating multi-band, raw datasets with PhoSim - Creating a data repository for the data - Creating an astrometry_net_data repository - Processing raw files through processCdc (including ISR, measurement, etc.) - Creating a catalog of measurements - Visualizing the output using existing tools It is possible to work initially with a limited set of data (a raft, or CCD), and to start with eImages rather than raw; the example can be elaborated in due course.",10 DM-472,"Create an iPython Notebook visualization of the LSST Demo Data","Develop an iPython Notebook to illustrate the processing and results of the LSST Demo. This can be used as both a basic tutorial for the Stack, and a how-to for creating additional visualizations for DM. The visualization should include the source catalog, and possibly also one or more of the processed images. ",4 DM-473,"Prioritize and define the backlog for Summer 2014","At the DMLT meeting in Seattle, finish the backlog prioritization for Summer 2014 cycle. Definition of Done: * All epics and major stories defined for Summer 2014. * Rank-ordered in terms of priority. * Leadership teem agreement on their prioritization and backlog. * Teams identified to execute the stories, and stories labeled accordingly. ",2 DM-474,"Document how to create an astrometry_net_data repository","Document how to create astrometry_net_data index files for an arbitrary astronomical dataset, as well as how to create a repository of such data to support processing with the LSST Stack. It must be possible to limit the area of sky coverage in the index files to that appropriate for the images that the user wishes to process. ",6 DM-488,"Make JIRA notification e-mail more useful","From HipChat/Data Management: [12:09] Mario Juric: @jbosch @KTL @KSK Could you double-check if any of you got an e-mail from Jira on Saturday (Apr 12th) re issue DM-78 (I made you reviewers, but it looks like you weren't notified)? [12:10] K-T Lim: I don't recall and can't determine now; it would have been deleted (irrevocably). [12:10] Simon Krughoff: I did get an email. [12:10] Jim Bosch: @mjuric, ah, it appears that I actually did. The fact that I was a reviewer was just buried, and I didn't notice it. [12:10] Simon Krughoff: I must have missed that I was a reviewer. [12:10] Mario Juric: OK, thanks! That gives me not one, but two useful data points (#1 -- emails work, #2 -- they're useless :) ). [12:12] Simon Krughoff: I'm not sure why they are useless. The emails from trac were a very important part of my workflow as far as being notified of review responsibility goes. Maybe it's just the volume from Jira. [12:14] Jim Bosch: Yeah, same here. Though the volume from JIRA hasn't been so bad, so I don't think that's it. Maybe my brain just has to get used to the new email format. [12:14] K-T Lim: (In my case, I'm mostly paying attention to the RSS feed although the mailbox serves as a backup.) [12:22] Robert Lupton: One of the things that made gnats a good bug tracker was that the emails contained the right amount of information (I did have source code...), and trac was pretty good too when we tuned it; bugzilla always used to be awful. I bet we can fiddle with Jira to make its mail more useful; I don't just mean filtering what it sends, but making sure that each email is self contained, but not too long",1 DM-495,"Build Base DMCS Archiver Command Receiver","Base DMCS Archiver The Archiver receives commands from the OCS, and performs the following actions: * init - Move from OFFLINE mode to IDLE state. * configure - (arguments: set of computers to use, the software to be executed, and parameters to control that software). 1) Verify format and accept command; 2) change to disable mode as if a disable command had been received; 3) install configuration; 4) return success to the OCS. On failure (illegal configuration, cannot be installed properly), a command error with the failure reason is sent to the OCS. prerequisite for configure command - sufficient replicator/distributor pairs are available. Note that after configure, the Archiver remains disabled. * enable - Subscribe to the startIntegration event. * disable - Unsubscribe to the startIntegration event. Note that this does NOT terminate any replicator jobs which are already executing. * release - Unsubscribe to the startIntegration event, and go into OFFLINE state. * abort - if received during a configure command, causes this component to go into the ERROR state with no configuration. If received at any other time, the system transitions into an ERROR state, but nothing stops that was already in progress. * reset - Unsubscribe to the startIntegration event, go into IDLE state, with no configuration",6 DM-502,"Build Base DMCS Catch-Up Archiver Command Receiver","Base DMCS Catch-Up Archiver The Catch-Up Archiver receives commands from the OCS, and performs the following actions: * init - Move from OFFLINE mode to IDLE state. * configure - (arguments: set of computers to use, the software to be executed, and parameters to control that software). 1) Verify format and accept command; 2) change to disable mode as if a disable command had been received; 3) install configuration; 4) return success to the OCS. On failure (illegal configuration, cannot be installed properly), a command error with the failure reason is sent to the OCS. Prerequisite for configure command - sufficient catch-up-dedicated replicator/distributor pairs are available. Note that after configure, the Archiver remains disabled. * enable - 1) Allows Catch-Up Archiver to scan for unarchived images to be handled; 2) Enables the Orchestration Manager to schedule image archive jobs. * disable - 1) Stop scanning for unarchived images; 2) Tell the Orchestration Manager to stop scheduling any new image archive jobs. * release - Unsubscribe to the startIntegration event, and go into OFFLINE state. * abort - if received during a configure command, causes this component to go into the ERROR state with no configuration. If received at any other time, the system transitions into an ERROR state, but nothing stops that was already in progress. * reset - Unsubscribe to the startIntegration event, go into IDLE state, with no configuration",6 DM-503,"Build Base DMCS EFD Replicator Command Receiver","Base DMCS EFD Replicator The EFD Replicator receives commands from the OCS, and performs the following actions: * init - Move from OFFLINE mode to IDLE state. * configure - (arguments: set of computers to use, the software to be executed, and parameters to control that software). 1) Verify format and accept command; 2) change to disable mode as if a disable command had been received; 3) install configuration; 4) return success to the OCS. On failure (illegal configuration, cannot be installed properly), a command error with the failure reason is sent to the OCS. Prerequisite for configure command - communication with the US DAC EFD replica is possible. Note that after configure, the EFD Replicator remains disabled. * enable - Causes the Base DMCS to enable the US DAC EFD replica to be a slave to the Chilean DAC EFD replica. * disable - Causes the Base DMCS to disable the slave operation of the US DAC EFD replica. * release - Unsubscribe to the startIntegration event, and go into OFFLINE state. * abort - if received during a configure command, causes this component to go into the ERROR state with no configuration. If received at any other time, the system transitions into an ERROR state, but nothing stops that was already in progress. * reset - Unsubscribe to the startIntegration event, go into IDLE state, with no configuration",6 DM-504,"Build Base DMCS Alert Production Cluster Command Receiver","Base DMCS Alert Production Cluster The Alert Production Cluster receives commands from the OCS, and performs the following actions: * init - Move from OFFLINE mode to IDLE state. * configure - (arguments: set of computers to use, the software to be executed, and parameters to control that software). 1) Verify format and accept command; 2) change to disable mode as if a disable command had been received; 3) install configuration; 4) return success to the OCS. On failure (illegal configuration, cannot be installed properly), a command error with the failure reason is sent to the OCS. Prerequisite for configure command - sufficient workers are available. Note that after configure, the Alter Production Cluster remains disabled. * enable - Causes the Base DMCS to subscribe to the “nextVisit” topic in normal science mode; another event may be subscribed to in calibration or engineering mode. * disable - Causes the Base DMCS to unsubscribe from the “nextVisit” topic. It does NOT terminate any worker jobs already executing. In particular, the processing for the current visit (not just exposure) will normally complete. * release - Unsubscribe to the startIntegration event, and go into OFFLINE state. * abort - if received during a configure command, causes this component to go into the ERROR state with no configuration. If received at any other time, the system transitions into an ERROR state, but nothing stops that was already in progress. * reset - Unsubscribe to the startIntegration event, go into IDLE state, with no configuration",6 DM-505,"improve initialization of kvMap in testQueryAnalysis","Build the kvMap at build-time and embed it into the executable. (this was brought up in DM-225)",1 DM-506,"improve generating kvMap in testFacade.cc","Generating the kvmap file, and pasting it into a string inside the test program. (this was brought up in DM-225)",1 DM-508,"shorten internal names in zookeeper","rename DATABASE_PARTITIONING to PARTITIONING rename DATABASES to DBS",2 DM-509,"rename ""dbGroup"" to ""storageClass"" in CSS metadata","It is meant to be used to indicate L1, L2, L3... At Qserv design week we decided to rename it (original plan was to remove it all together) ",1 DM-510,"Tweak metadata structure for driving table and secondary index","There seem to be confusion about driving table and secondary index. At the moment in zookeeper structure we have {code} /DATABASES//objIdIndex /DATABASES//TABLES//partitioning/secIndexColName /DATABASES//TABLES//partitioning/drivingTable /DATABASES//TABLES//partitioning/latColName /DATABASES//TABLES//partitioning/lonColName /DATABASES//TABLES//partitioning/keyColName {code} Issues to think about: * we can't call it objIdIndex, it is too lsst-specific. * drivingTable and keyColName - perhaps these should be at database level, which means we would only allow one drivingTable and one secondary index per database? * or, maybe instead of database level, it is a partitioning parameter? Note that two databases might use different name for secondary index or driving table, yet they might be joinable. That argues for introducing a new group, something like /DATABASE/partitioning in addition to /DATABASE_PARTITIONING. * consider renaming drivingTable to keyTable * do we really need secIndexColName and keyColName? Can't we get rid of one, and rename to keyColName? ",2 DM-511,"rework ubuntu.patch",NULL,1 DM-512,"Generalizing data chunking (n-level chunking rather than stripes/subStripes)","It'd be cleaner to use numbering (e.g. 0 for no chunking, 1 for chunks, 2 for subchunks etc) instead of ""chunks"", ""subchunks"" throughout qserv code. This might also be true in partitioner, where flags like -S and -s etc are not entirely obvious. This came up in the review of CSS, see DM-225.",3 DM-513,"fix threading issues in CSS watcher","Fix problems with threads in watcher.py brought up in DM-225 by Serge: * A thread per database doesn't scale * There is a thread leak when a database is deleted * There is another design problem, in that each database thread looks like it is holding on to the same lsst.db.Db instance under the hood. I don't remember any consideration for thread safety from the lsst.db code when I reviewed it. Note for one that it is not safe to use a MySQL connection simultaneously from multiple threads (and I seem to recall that you are caching a connection inside Db instances). In practice, even the Python GIL may not save you, since calls into C code (i.e. the mysql client library) may very well release it.",8 DM-514,"Switch to the ""czar"" name consistently","1) Change lsst.qserv.master to lsst.qserv.czar in the czar module. 2) Rename masterLib to czarLib 3) Rename startQserv to startCzar ",4 DM-516,"Fix race condition when creating db (and elsewhere?) in client/qserv_admin_impl.py","This came up in the review of CSS, see DM-225: QservAdminImpl: so, if there's a problem while inserting the various keys for a database, and another exception during cleanup, you can end up with a partially constructed database. How do you plan on handling this? This is perhaps another argument for trying to batch metadata more - if it's all in a single key, this worry goes away (at least in this particular case). The createDb implementations seem racy. Consider what happens if 2 admins try to create the same database D. Suppose admin A does the _dbExists(D) check first; it returns false. Then admin B comes along, and does the same check, which returns false again. Let's further suppose that admin B successfully issues all the key creates for D without being interrupted by A. Back to A: when A tries to create all the keys required for D, an exception will be raised immediately since D exists. At this point A will dutifully try to clean up (to avoid leaving around a partially constructed D), deleting D as created by B. End result: D was successfully created, no explicit drop was issued, but D does not exist. Database drops can also race with table creates: zookeeper doesn't have a native recursive delete. In other words, it is perfectly possible for an admin A to issue a database drop while someone else is issuing a table create. The recursive delete might descend into the table ""directory"" while the create is populating it with child znodes. The recursive delete will get an incomplete list of children, remove them, and finally fail to remove the table directory because the table create has since added more children, and it is illegal to remove a non-empty directory. At this stage, the drop will report success even though it has failed (that's the recursive delete behavior that kazoo gives you), and the table create will report success. But the database will still exist, and table keys may be partially present in zookeeper.",10 DM-517,"qserv_admin needs to deal with uncommon names/characters","qserv/admin/bin/qserv_admin.py needs to deal with strange names (e.g. with embedded spaces or semi-colons), or at minimum, catch and forbid them.",3 DM-518,"Rework exceptions in qserv client","There is a bunch of (I think) unnecessary translation from KvException to QservAdmException. Can't you just handle printing KvException in CommandParser.receiveCommands(), and get rid of the CSSERR error code? (This is in /admin/bin/qserv-admin.py) (this came up in DM-225)",1 DM-519,"rethink configuration for client","_fetchOptionsFromConfigFile in client/qserv_admin.py:: If you are going to use the ConfigParser library, please use SafeConfigParser (or RawConfigParser if you don't need string interpolation). But anyway, I'm not sure the INI file format is the right one to use here, as it forces the use of sections in parameter files, which doesn't make much sense (I notice your code just throws away the section names when reading parameter files). I found myself wishing that the admin client just had syntax for the various options, rather than relying on more or less undocumented parameter files. (this came up in DM-225)",3 DM-520,"Remove old partitioner/ loader and duplicator","Once Fabrice has migrated the integrated tests towards using the new partitioner and duplicator, we should delete the old partitioner/duplicator (in {{client/examples}}).",1 DM-521,"Confusing error message (non-existing column referenced)","A query that references non existing column for non-partitioned table results in a confusing message: ""read failed for chunk(s): 1234567890"". To repeat, run something like {code} SELECT whatever FROM ; {code} Similar error occurs when we try to reference non-existing table, try something like: {code} SELECT sce.filterName FROM StrangeTable AS s, Science_Ccd_Exposure AS sce WHERE (s.scienceCcdExposureId = sce.scienceCcdExposureId); {code} ",5 DM-522,"gracefully handle misconfigured scons","When I try to run scons using the latest master (89aaa6), it fails with {code} scons: Reading SConscript files ... AttributeError: 'NoneType' object has no attribute 'rfind': File ""/home/becla/cssProto/qserv_css6/SConstruct"", line 17: state.init(src_dir) File ""/home/becla/cssProto/qserv_css6/site_scons/state.py"", line 161: _initEnvironment(src_dir) File ""/home/becla/cssProto/qserv_css6/site_scons/state.py"", line 128: _initVariables(src_dir) File ""/home/becla/cssProto/qserv_css6/site_scons/state.py"", line 89: (PathVariable('XROOTD_DIR', 'xrootd install dir', _findPrefix(""XROOTD"", ""xrootd""), PathVariable.PathIsDir)), File ""/home/becla/cssProto/qserv_css6/site_scons/state.py"", line 53: (binpath, binname) = os.path.split(binFullPath) File ""/usr/lib/python2.7/posixpath.py"", line 83: i = p.rfind('/') + 1 {code} The scripts should check that requires variables are not set, and print appropriate error (and ideally, suggest how to fix it) ",1 DM-527,"make Image construction robust against integer overflow","I just fixed a bug on the HSC side (DM-523) in which integer overflow in the multiplication of width and height in image construction caused problems. We should backport this fix to LSST.",1 DM-530,"Table column names in new parser","Running tests (qserv-testdata.sh) on pre-loaded data I have observed that many test fail for the only reason that the column names in the dumped query results are different between mysql and qserv. Here is an example of query reqult returned from mysql: {code} mysql> SELECT sce.filterName, sce.field, sce.camcol, sce.run FROM Science_Ccd_Exposure AS sce WHERE sce.filterName = 'g' AND sce.field = 670 AND sce.camcol = 2 AND sce.run = 7202; +------------+-------+--------+------+ | filterName | field | camcol | run | +------------+-------+--------+------+ | g | 670 | 2 | 7202 | +------------+-------+--------+------+ {code} and this is the same query processed by qserv: {code} mysql> SELECT sce.filterName, sce.field, sce.camcol, sce.run FROM Science_Ccd_Exposure AS sce WHERE sce.filterName = 'g' AND sce.field = 670 AND sce.camcol = 2 AND sce.run = 7202; +----------+----------+----------+----------+ | QS1_PASS | QS2_PASS | QS3_PASS | QS4_PASS | +----------+----------+----------+----------+ | g | 670 | 2 | 7202 | +----------+----------+----------+----------+ {code} We discussed this already with Daniel yesterday and at qserv meeting today, here I just want to collect what we know so far so that we can return to this again later. As Daniel explained to me this is the result of the new parser assigning aliases to the columns which do not define aliases for themselves. This helps with tracking query proceeding through the processing pipeline. Daniel's observation is that different database engines may assign different names to result columns (or some may not even assign any names), there is no standard in that respect so there is no point in trying to follow what one particular implementation does. Additionally there are issues with conflicting column names and names which are complex expressions. Difference in column names breaks our tests which dump complete results including table header. The tests could be fixed easily, we could just ignore table headers when dumping the data. More interesting issue is that there may be use cases for better compatibility between mysql and qserv including result column naming. In particular standard Python mysql interface allows one to use column names to retrieve values from queiry result. If qserv assigns arbitrary aliases to the columns it may confuse this kind of clients. This issue depends very much on what kind of API qserv is going to provide to clients. If mysql (wire-level) protocol is going to be the main API (which would allow all kinds of mysql clients to talk to qserv directly) then we should probably think more about compatibility with mysql. OTOH if we decide to provide our own API then this may not be an issue at all (but we still need to fix current test setup which is based on mysql). We probably should discuss API question at our dev meeting.",5 DM-531,"Qserv returns error table instead of error code","Running the tests on pre-loaded data I noticed that for some queries qserv returns result which does not look like it is related in any way to the query - result column names are different from the columns in the query (different in a different way from DM-530). Here is an example of query and result produced: {code} mysql> SELECT sce.filterName, sce.field, sce.camcol, sce.run FROM Science_Ccd_Exposure AS sce WHERE sce.filterName like '%' AND sce.field = 535 AND sce.camcol like '%' AND sce.run = 94; +---------+------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+ | chunkId | code | message | timeStamp | +---------+------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+ | -1 | 0 | NULL | 1.39779e+09 | | -1 | 100 | Dispatch Query. | 1.39779e+09 | | 32767 | 1200 | Query Added: url=xroot://qsmaster@127.0.0.1:1094//q/LSST/1234567890, savePath=/dev/shm/qserv-salnikov-80df10ee4ed55693702f55021486cd45647c3a58ce549e7c826d9626/7_1234567890_0 | 1.39779e+09 | | 32767 | 1300 | Query Written. | 1.39779e+09 | | 32767 | 1400 | Results Read. | 1.39779e+09 | | 32767 | 1500 | Results Merged. | 1.39779e+09 | | 32767 | 1600 | Query Resources Erased. | 1.39779e+09 | | 32767 | 2000 | Query Finalized. | 1.39779e+09 | +---------+------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+ {code} Daniel and Bill explained to me what is happening there - when query results in an error instead of returning mysql error condition (which is an error code plus some text) proxy produces a diagnostic result which is a table (above) containing some info which could be useful to diagnose the issue. This feature looks potentially useful but it may also be confusing for clients like me who are not aware of this feature. For regular mysql client it may be actually harder to intercept errors because one would need to analyze returned table to understand that error condition happened. It may also be ambiguous in a sense that the legitimate query could produce result with the same column names. Like in DM-530 this issue is tied to a question what kind of API we want to provide to qserv clients. If mysql wire-level protocol is going to be our main API then we should probably try to be more mysql-compatible. OTOH if we are going to hide everything behind our own API then we may have more freedom in re-defining what kind of result error condition produces.",4 DM-533,"transfer multiband processing changes from HSC","We have a new multi-band processing scheme for coadds on the HSC side, encompassing changes in afw, meas_deblender, and pipe_tasks. These should be transferred to the LSST side, with RFCs for any backwards incompatible changes. Changes to the Footprint classes may only be temporary, as we plan to refactor those classes soon anyway, but they're still worth doing now to support the higher-level changes.",8 DM-536,"Move HSC issues to hsc-jira.astro.princeton.edu",NULL,10 DM-545,"ensure pipe_tasks, obs*, and other packages are compatible with meas_base","The switch to meas_base will involve changing the names of most measurement algorithms, as we're using a new naming convention that provides more traceability. This will break downstream code that: - uses field names instead of slots to access measurements - reads or modifies the list of configured-to-run algorithms. Whenever possible, we should fix the former by converting them to use slots, as this will automatically provide backwards compatibility. I'm not yet sure how to handle the latter; the easiest solution would be to give up on full backwards-compatibility with meas_algorithms, but we might be able to find some way to make the old names aliases to the new ones during the deprecation period. ",12 DM-546,"scons rebuilds targets without changes","I'm seeing something strange when I run scons from current master - running 'scons install' after 'scons build' re-compiles several C++ files even though nothing has changed between these two runs: {code:bash} $ scons build scons: Reading SConscript files ... ... scons: Building targets ... scons: `build' is up to date. scons: done building targets. $ scons install scons: Reading SConscript files ... ... scons: Building targets ... swig -o build/czar/masterLib_wrap.cc -Ibuild -I/usr/include/python2.6 -python -c++ -Iinclude build/czar/masterLib.i g++ -o build/czar/masterLib_wrap.os -c -g -fPIC -D_FILE_OFFSET_BITS=64 -fPIC -I/u2/salnikov/STACK/stack/Linux64/protobuf/2.4.1/include -I/u2/salnikov/STACK/stack/Linux64/xrootd/qs5/include/xrootd -I/u2/salnikov/STACK/stack/Linux64/mysql/5.1.65/include -Ibuild -I/usr/include/python2.6 build/czar/masterLib_wrap.cc g++ -o build/control/AsyncQueryManager.os -c -g -fPIC -D_FILE_OFFSET_BITS=64 -fPIC -I/u2/salnikov/STACK/stack/Linux64/protobuf/2.4.1/include -I/u2/salnikov/STACK/stack/Linux64/xrootd/qs5/include/xrootd -I/u2/salnikov/STACK/stack/Linux64/mysql/5.1.65/include -Ibuild -I/usr/include/python2.6 build/control/AsyncQueryManager.cc g++ -o build/control/dispatcher.os -c -g -fPIC -D_FILE_OFFSET_BITS=64 -fPIC -I/u2/salnikov/STACK/stack/Linux64/protobuf/2.4.1/include -I/u2/salnikov/STACK/stack/Linux64/xrootd/qs5/include/xrootd -I/u2/salnikov/STACK/stack/Linux64/mysql/5.1.65/include -Ibuild -I/usr/include/python2.6 build/control/dispatcher.cc g++ -o build/control/thread.os -c -g -fPIC -D_FILE_OFFSET_BITS=64 -fPIC -I/u2/salnikov/STACK/stack/Linux64/protobuf/2.4.1/include -I/u2/salnikov/STACK/stack/Linux64/xrootd/qs5/include/xrootd -I/u2/salnikov/STACK/stack/Linux64/mysql/5.1.65/include -Ibuild -I/usr/include/python2.6 build/control/thread.cc g++ -o build/merger/TableMerger.os -c -g -fPIC -D_FILE_OFFSET_BITS=64 -fPIC -I/u2/salnikov/STACK/stack/Linux64/protobuf/2.4.1/include -I/u2/salnikov/STACK/stack/Linux64/xrootd/qs5/include/xrootd -I/u2/salnikov/STACK/stack/Linux64/mysql/5.1.65/include -Ibuild -I/usr/include/python2.6 build/merger/TableMerger.cc scons: `install' is up to date. scons: done building targets. {code} This is kind of unexpected, or at least I can't understand now why it happens. Trying to run with --debug=explain shows that some dependencies have disappeared and in some dependencies order is different. No clue yet what that means and how it could happen. Need to study our scons scripts to understand what is going on.",3 DM-548,"rearchitect Qserv to fix dependencies between modules in qserv/core","I looked at includes in ""core/modules"", here is a summary. * "":"" indicates a dependency * global, log, util, wbase, wlog and xrootd are low level, it is an easy, I didn't bother showing them below * ""!!!"" indicates circular dependency * lower-level modules first {core} css: global: mysql: obsolete: proto: sql: mysql wconfig: sql, mysql wpublish: sql, wconfig !!! wsched <--> wcontrol !!! wcontrol <--> wdb wsched: proto, wcontrol wcontrol: mysql, sql, wsched, wdb, wcontrol, obsolete, proto wdb: sql, mysql, proto, wconfig, wcontrol xrdfs: wpublish, wdb, wconfig, sql, obsolete, wcontrol merger: sql, xrdfs xrdoss: obsolete, wpublish, xrdfs !!! control <--> qdisp !!! qana <--> qproc !!! qana <--> query !!! parser --> query -->qana --> parser !!! query --> qana --> qproc --> query control: css, merger, obsolete, qdisp, qproc, query qdisp: control qana: css, parser, qproc, query qproc: css, merger, parser, proto, qana, qdisp, query query: css, qana, qdisp parser: query {core} ",10 DM-554,"Base DMCS and Replicator Interaction for Simulator - v1","Create the Base DMCS and Replicator processes that demonstrate Send startIntegration Send startReadout Assume: Archiver and Alert Production Cluster already enabled, replicators already registered in fully-operational pool. Demonstrate: Sending startIntegration event to the Base DMCS. Replicator jobs submission Replicator jobs subscribe to startReadout Sending the startReadout event to the replicator jobs. Replicator jobs receive startReadout Replicator jobs pull data ",10 DM-559,"clean up include <> --> """" for third party includes","According to our coding standard 4.15: https://dev.lsstcorp.org/trac/wiki/C%2B%2BStandard/Files#a4-15.OnlysystemincludefilepathsSHALLbedelimitedwith we should be using """" for boost, but in quite a few places we do not: {code} grep 'include SELECT objectId FROM Object WHERE qserv_areaspec_box(0, 0, 3, 10) ORDER BY objectId; +-----------------+ | objectId | +-----------------+ | 417857368235490 | | 417861663199589 | | 417865958163688 | | 420949744686724 | | 420954039650823 | | 424042121137958 | | 424046416102057 | | 427134497589192 | | 430226874040426 | +-----------------+ 9 rows in set (1.27 sec) mysql> SELECT objectId FROM Object WHERE qserv_areaspec_box(0, 0, 3, 10) ORDER BY objectId; +-----------------+ | objectId | +-----------------+ | 424042121137958 | | 424046416102057 | | 427134497589192 | | 430226874040426 | | 417857368235490 | | 417861663199589 | | 417865958163688 | | 420949744686724 | | 420954039650823 | +-----------------+ 9 rows in set (1.24 sec) {code} This was done with testdata/case01 data, let me know if you need to load that data. Also, using case03 data, e.g. {code} SELECT distinct run, field FROM Science_Ccd_Exposure WHERE run = 94 AND field = 535; {code} returns 6 rows in qserv (vs 1 in mysql). The full list of DISTINCT failures is (all with testdata/case03): - 0002_fetchRunAndFieldById.txt - 0021_selectScienceCCDExposure.txt - 0030_selectScienceCCDExposureByRunField.txt ",1 DM-627,"Switch to using new partitioner, loader","Integrated tests procedure has to rely on new loader",4 DM-630,"Non-partitioned table query returns duplicated rows","Running automated test I noticed that a query on non-partitioned table returns multiple copies of the same row, one copy per chunk. Here is example: {code} mysql> SELECT offset, mjdRef, drift FROM LeapSeconds where offset = 10; +--------+--------+-------+ | offset | mjdRef | drift | +--------+--------+-------+ | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | +--------+--------+-------+ 13 rows in set (5.62 sec) {code} Czar log file shows that it correctly finds that table is non-chunked but sends query to each chunk anyway: {code} 20140502 16:22:08.745081 0x3172430 INF *** KvInterfaceImplZoo::exist(), key: /DATABASES/LSST/TABLES/LeapSeconds/partitioning 20140502 16:22:08.745735 0x3172430 INF *** LSST.LeapSeconds is NOT chunked. 20140502 16:22:08.745762 0x3172430 INF *** KvInterfaceImplZoo::get2(), key: /DATABASES/LSST/TABLES/LeapSeconds/partitioning/subChunks 20140502 16:22:08.746393 0x3172430 INF *** LSST.LeapSeconds is NOT subchunked. 20140502 16:22:08.746409 0x3172430 INF getChunkLevel returns 0 ..... 20140502 16:22:08.757832 0x3172430 INF Using 85 stripes and 12 substripes. 20140502 16:22:08.775586 0x3172430 INF Using /usr/local/home/salnikov/dm-613/build/dist/etc/emptyChunks.txt as default empty chunks file. 20140502 16:22:08.791559 0x3172430 INF empty_LSST.txt not found while loading empty chunks file. 20140502 16:22:08.791592 0x3172430 ERR Couldn't find empty_LSST.txt, using /usr/local/home/salnikov/dm-613/build/dist/etc/emptyChunks.txt. 20140502 16:22:08.891239 0x7fbb28003660 INF QuerySession::_buildChunkQueries() : Non-subchunked 20140502 16:22:08.891498 0x7fbb28003660 INF Msg cid=6630 with size=153 20140502 16:22:08.891682 0x7fbb28003660 INF Added query id=6630 url=xroot://qsmaster@127.0.0.1:1094//q/LSST/6630 with save /dev/shm/qserv-salnikov-b93a8b7cca1128f50fa5531feb93f8f24a185f162d36c10ee76b8dca/1_6630_0 20140502 16:22:08.891694 0x7fbb28003660 INF Opening xroot://qsmaster@127.0.0.1:1094//q/LSST/6630 20140502 16:22:08.891705 0x7fbb28003660 INF QuerySession::_buildChunkQueries() : Non-subchunked 20140502 16:22:08.891882 0x7fbb28003660 INF Msg cid=6631 with size=153 20140502 16:22:08.892077 0x7fbb28003660 INF Added query id=6631 url=xroot://qsmaster@127.0.0.1:1094//q/LSST/6631 with save /dev/shm/qserv-salnikov-b93a8b7cca1128f50fa5531feb93f8f24a185f162d36c10ee76b8dca/1_6631_0 20140502 16:22:08.892087 0x7fbb28003660 INF Opening xroot://qsmaster@127.0.0.1:1094//q/LSST/6631 20140502 16:22:08.892097 0x7fbb28003660 INF QuerySession::_buildChunkQueries() : Non-subchunked 20140502 16:22:08.892275 0x7fbb28003660 INF Msg cid=6800 with size=153 20140502 16:22:08.892462 0x7fbb28003660 INF Added query id=6800 url=xroot://qsmaster@127.0.0.1:1094//q/LSST/6800 with save /dev/shm/qserv-salnikov-b93a8b7cca1128f50fa5531feb93f8f24a185f162d36c10ee76b8dca/1_6800_0 ... {code} Looking at the code together with Daniel we found that at the Python level (czar/app.py) the code that dispatches query does not check for chunkLevel, this is likely why this happens. The code to look at is in {{InbandQueryAction._applyConstraints()}} method.",5 DM-631,"Define C++ API for C++ Geometry",NULL,10 DM-633,"Query sessions are never destroyed","Please see DM-625, when I run say 10 ""select count(*) from LSST.Object"" queries, for each query a new AsyncQueryManager is created in dispatcher, but the sessions are never destroyed.",3 DM-635,"admin/tests/test_qservAdminImpl.py has hardcoded connection info",NULL,1 DM-637,"complexity of eups dependencies relationships for db package","Hello, I'm currently trying to use the very last version of db package (the one which relies on sconsUtils), but, in order to make it works with Qserv, I had to introduce next update : {code:bash} fjammes@clrlsstwn02-vm:~/src/qserv-packager/dist/dependencies/db (master) $ git diff HEAD~1 diff --git a/ups/db.cfg b/ups/db.cfg index e1ae31b..a469061 100644 --- a/ups/db.cfg +++ b/ups/db.cfg @@ -3,7 +3,7 @@ import lsst.sconsUtils dependencies = { - ""required"": [""mysqlclient"", ], + ""required"": [""mysql"", ], } config = lsst.sconsUtils.Configuration( diff --git a/ups/db.table b/ups/db.table index 8c8d831..9e770a3 100644 --- a/ups/db.table +++ b/ups/db.table @@ -1,5 +1,5 @@ setupRequired(python) -setupRequired(mysqlclient) +setupRequired(mysql) setupRequired(mysqlpython) setupRequired(sconsUtils) {code} Is there a solution to describe in eups that mysqlclient is included in mysql ? Thanks, Fabrice",1 DM-644,"update overview docs to clarify roles of meas_multifit and shapelet packages","From the review of DM-17: {quote} It was not obvious how responsibility is split between {{meas_extensions_multishapelet}}, {{shapelet}}, and {{meas_multifit}}. Shapelet could use an {{overview.dox}} file. {quote} {{meas_extensions_multiShapelet}} is on its way out, so we'll wait until that's done and then document the relationship between meas_multifit and shapelet.",2 DM-645,"meas_base plugin for sampling-based galaxy fitter","In addition to a CModel plugin for galaxy photometry (DM-240), we should create a plugin to do sampling-based galaxy fitting, using the optimizer as a starting point and the existing AdaptiveImportanceSampler class to do most of the work. A major blocker for this is the fact that the {{SourceTable/SourceRecord}} don't currently provide any way to save multiple samples per object. This issue may get worked on before it falls into an official sprint, as it's something I want to on my 20% time.",8 DM-646,"Implement DISTINCT aggregate in qserv","It looks like DISTINCT aggregate is not supported yet in qserv. Daniel told me that this should be relatively straightforward to add. Adding this ticket so that we do not forget it.",2 DM-647,"support samples and other many-to-one outputs in SourceRecord","For DM-645, I need to be able to save multiple samples (each a parameter vector and possibly a weight) with each {{SourceRecord}}, with the number of samples possibly differing from {{SourceRecord}} to {{SourceRecord}}. I'll also want to save Mixture distributions, which can also be easily represented as a sequence-of-tuples of values to be associated with a record. Since we don't want to hard-code the outputs of a particular plugin into {{SourceRecord}} (which is what I've done with the {{ModelFitRecord}} class in meas_multifit, which I'd like to remove), I think this means we want a more general way of allowing algorithms to define many-to-one tables to be associated with each source. While few algorithms will likely want to save samples, this plugin may not be the only one, and it's likely a general many-to-one feature could be used by other algorithms to save diagnostics when configured to do so. I'm worried this is another case where afw::table may be intruding on a feature best left to a true RDBMS, but I don't like the idea of tying an RDBMS directly to the source measurement framework either. I'm open to other ideas, if anyone has one. It's also worth pointing out that because this is blocking DM-645, which is important for my 20%-time science work, I'd like to get moving on this even before it becomes a real priority for LSST. But I'd still like to get some design feedback on this.",20 DM-648,"Add support for running unit tests in scons","Add code in scons that runs unit tests for Qserv.",5 DM-649,"framework for documenting ""how to run qserv""","We need to have permanent location for how-to-run-qserv, currently we keep it in https://dev.lsstcorp.org/trac/wiki/db/Qserv/RedesignFY2014/Hackathon2/howToRunQserv. It should be in the code repo. Need to decide on how we format it, and need to expose it on our Qserv trac/confluence pages ",3 DM-650,"Migrate database trac pages to Confluence","Need to migrate Qserv trac pages to Confluence.",10 DM-653,"Run baseline HTCondor ClassAds Scenarios","Run initial HTCondor ClassAds Scenarios to verify that the implementation for utilizing Rank to place Jobs near data is operating as anticipated. The main test of the baseline scenarios is to verify that, e.g., a job for a CCD that has calibrations advertised for a particular slot/node will consistency be executed within that location for appropriate Rank expression. This is to occur even when other open slots are always available (e.g., 4 CCDs, 5 slots). ",4 DM-654,"Run ""single slow worker"" HTCondor ClassAds Scenario","We run and study a ""single slow worker"" HTCondor ClassAds Scenario. The scenario is a perturbation of the baseline HTCondor ClassAds Scenario. In the baseline, Jobs for a given CCD are consistently pinned to a slot/node that advertises the presence of associated data files/calibration files for that CCD. A baseline run may proceed, for example, with jobs for 4 CCDs repeating executing within same HTCondor slot on a node (even when spare processing slots are readily available.). The baseline is observed to be quite stable, as the pool is empty each time a wave of jobs is submitted. In the ""single slow worker"" scenario, we cause one of the jobs for a chosen CCD to stall (mocking up a slow file transfer, lengthy computation in an algorithm, etc), such that the pool is not empty at the the submission time for a wave of jobs. We seek to observe how Rank places jobs in this scenario, and work to assign Rank (especially for spare slots) in an optimal way so as to minimize file transfers. ",4 DM-655,"unknown column derails Qserv","Running a query that references invalid column hangs Qserv, it looks like the query is never squashed. For example, I run a query: {code} SELECT distinct run, field FROM Science_Ccd_Exposure WHERE run = 94 AND field = 535; {code} on the pt1.1 data set Corresponding log from xrootd: {code} Foreman:>>Unknown column 'run' in 'field list' Unable to execute query: CREATE TABLE r_7d8e5a45a6aba94de141cf1d8ab510c89_3598_0 SELECT run,field FROM LSST.Science_Ccd_Exposure \ AS QST_1_ WHERE run=94 AND field=535; <<---Error with piece 0 complete (size=1). Foreman:TIMING,q_f99cQueryExecFinish,1399667470 Foreman:Broken! ,q_f99cQueryExec---Unknown column 'run' in 'field list' Unable to execute query: CREATE TABLE r_7d8e5a45a6aba94de141cf1d8ab510c89_3598_0 SELECT run,field FROM L\ SST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535; (during QueryExec) QueryFragment: CREATE TABLE r_7d8e5a45a6aba94de141cf1d8ab510c89_3598_0 SELECT run,field FROM LSST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535; Foreman:Fail QueryExec phase for q_f99c: Unknown column 'run' in 'field list' Unable to execute query: CREATE TABLE r_7d8e5a45a6aba94de141cf1d8ab510c89_3598_0 SELECT run,field \ FROM LSST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535; (during QueryExec) QueryFragment: CREATE TABLE r_7d8e5a45a6aba94de141cf1d8ab510c89_3598_0 SELECT run,field FROM LSST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535; Foreman:(FinishFail:0x7f26700055c0) Db = q_f99cc9cd5519d465e673119a84b5570a, dump = /usr/local/home/becla/qserv/1/qserv/build/dist/xrootd-run/result/f99cc9cd5519d465e673119a84b\ 5570a hash=f99cc9cd5519d465e673119a84b5570a Foreman:Finished task Task: msg: session=7 chunk=3598 db=LSST entry time=Fri May 9 15:31:10 2014 frag: q=SELECT run,field FROM LSST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535, sc= rt=r_7d8e5a45a6aba94de141cf1d8ab510c89_3598_0 ScanSched:ChunkDisk remove for 3598 : SELECT run,field FROM LSST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535 BlendSched:Completed: (3598)SELECT run,field FROM LSST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535 ScanSched:Completed: (3598)SELECT run,field FROM LSST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535 ScanSched:_getNextTasks(32)>->-> ScanSched:ChunkDisk busyness: no ScanSched:_getNextTasks <<<<< BlendSched:Blend trying other sched. GroupSched:_getNextTasks(4)>->-> GroupSched:_getNextTasks <<<<< 140509 15:31:27 13960 cms_Finder: Waiting for cms path /usr/local/home/becla/qserv/1/qserv/build/dist/tmp/worker/.olb/olbd.admin {code} and the log in czar has {code} 20140509 15:35:40.070654 0x7f4694003660 INF Still 1 in flight. {code} ",4 DM-658,"Database name used by integration tests should use their own dedicated database, not ""LSST""","Currently automated tests use the database ""LSST"", and they will go ahead and delete that database without any warning. This is far from ideal, we should have a more unique database name. How about something like qservAutoTest_.",9 DM-661,"Parser has inverted order for ""limit"" and ""order by""","{code} SELECT run FROM LSST.Science_Ccd_Exposure order by field limit 2 {code} Works in MySQL, fails in Qserv (ERROR 4120 (Proxy): Error executing query using qserv.) {code} SELECT run FROM LSST.Science_Ccd_Exposure limit 2 order by field {code} Works in Qserv, fails in MySQL (limit should be after order by) ",1 DM-664,"""out of range value"" message when running qserv-testdata (loader.py)","Fabrice I am getting ""out of range value"" when I run the qserv-testdata: Are you seeing that too? 2014-05-09 18:11:55,975 {/usr/local/home/becla/qserv/1/qserv/build/dist/lib/python/lsst/qserv/admin/commons.py:134} INFO stderr : /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'calib_detected' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'calib_psf_candidate' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'calib_psf_used' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_negative' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_badcentroid' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'centroid_sdss_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_edge' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_interpolated_any' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_interpolated_center' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_saturated_any' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_saturated_center' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_cr_any' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_cr_center' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'centroid_gaussian_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'centroid_naive_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'shape_sdss_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'shape_sdss_centroid_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'shape_sdss_flags_unweightedbad' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'shape_sdss_flags_unweighted' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'shape_sdss_flags_shift' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'shape_sdss_flags_maxiter' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_psf_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_psf_flags_psffactor' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_psf_flags_badcorr' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_naive_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_gaussian_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_gaussian_flags_psffactor' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_gaussian_flags_badcorr' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_sinc_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_psf_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_psf_flags_maxiter' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_psf_flags_tinystep' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_psf_flags_constraint_r' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_psf_flags_constraint_q' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flux_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flags_psffactor' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flags_badcorr' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flags_maxiter' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flags_tinystep' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flags_constraint_r' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flags_constraint_q' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flags_largearea' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flux_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flags_psffactor' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flags_badcorr' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flags_maxiter' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flags_tinystep' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flags_constraint_r' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flags_constraint_q' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flags_largearea' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_combo_flux_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_combo_flags_psffactor' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_combo_flags_badcorr' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'calib_detected' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'calib_psf_candidate' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'calib_psf_used' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_negative' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_badcentroid' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'centroid_sdss_flags' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_edge' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_interpolated_any' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_interpolated_center' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_saturated_any' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_saturated_center' at row 2 self.cursor.execute(stmt) ",2 DM-666,"partition package has to detect eups-related boost","partition package doesn't detect eups-related boost. This has to be fixed by using sconsUtils, or hand-made procedure. {code:bash} [fjammes@lsst-dev lsstsw]$ export LD_LIBRARY_PATH=""$LSSTSW/anaconda/lib:$LD_LIBRARY_PATH"" [fjammes@lsst-dev lsstsw]$ setup boost 1.55.0.1+1 [fjammes@lsst-dev lsstsw]$ rebuild partition partition: ok (0.5 sec). boost: ok (0.3 sec). python: ok (0.3 sec). scons: ok (0.4 sec). # BUILD ID: b49 python: master-gcbf93ab65b (already installed). scons: 2.1.0+8 (already installed). boost: 1.55.0.1+1 (already installed). partition: master-gf2ef2cf2dc ERROR (1 sec). *** error building product partition. *** exit code = 1 *** log is in /lsst/home/fjammes/src/lsstsw/build/partition/_build.log *** last few lines: ::::: scons: Reading SConscript files ... ::::: Checking for C++ library boost_system-mt... no ::::: Checking for C++ library boost_system... no ::::: Checking for C++ library boost_thread-mt... no ::::: Checking for C++ library boost_thread... no ::::: Checking for C++ library boost_filesystem-mt... no ::::: Checking for C++ library boost_filesystem... no ::::: Checking for C++ library boost_program_options-mt... no ::::: Checking for C++ library boost_program_options... no ::::: Missing required boost library! # BUILD b49 completed. {code}",3 DM-674,"fix handling of nested control objects","Work on the HSC side has revealed some problems with nested control objects being wrapped into config objects. This is a pull request for those changes (along with writing a unit test for some of them). Some (but not all of these changes) are part of Trac ticket #3163 (https://dev.lsstcorp.org/trac/ticket/3163), which I'll now close as a duplicate.",2 DM-675,"Citizen methods should be private and accessible only through a friend interface","The Citizen interface is useful, but it pollutes its derived classes with methods and attributes that can cause confusion later on (I've got a concrete example of that confusion that Perry and I just spent a few days tracking down - Citizen's {{getId()}} was being mistaken for {{SourceRecord.getId()}}). I think everything Citizen provides should be hidden and only accessible through a friend interface, e.g.: {code} afw::image::Image image(4, 5); daf::base::CitizenAccess::getId(image); {code} We should also make an effort to ensure that other aspects of Citizen's design don't affect derived classes, perhaps by prefixing an name that could be seen by derived classes with a ""Citizen"" prefix; see https://dev.lsstcorp.org/trac/ticket/2461.",4 DM-676,"Implement HTCondor dynamic classad solution for Slot based values","The HTCondor team will be updating their HOWTO for managing Slot based classads/dynamic classads set by a cron startd process. We currently have a technique for dynamic slot based values that is iinefficient from a negotiation perspective, and we will want to update to a more optimal approach that the HTCondor team plans to provide.",2 DM-677,"Develop monitoring for identifying Data processed on a Node/in a Slot","To understand the effectiveness with which we are mapping Jobs to Data, it is vital to monitor/record what data has been processed on a given Node, or within a given Slot on a Node. Under this issue we examine HTCondor monitoring standards like STARTD_HISTORY, as well as more custom implementation of blackboard type records via Job Update hooks to be executed on the execute node (along the lines of OWL.) ",4 DM-678,"Run HTCondor ClassAds Scenarios with heterogeneous data cache","The initial series of tests with HTCondor ClassAds work with jobs for individual ccds with a single file representing the data dependency for the job. In this issue we consider the management of multiple types of data dependencies that may have to be cached for jobs (calibrations, templates, catalogs of sources/objects, etc). ",4 DM-680,"Study ORDER BY support","We don't have a proper implementation of ORDER BY. Actually, to support ORDER BY properly, we really have to manage all the column names, so we would need to have an in-memory list of columns for the table in question. This is because the general case requires us to ORDER BY a column that may not exist in the select list. In this case, we must add it to the select list if it is not there (or apply *). The easiest solution is to only allow ORDER BY if the sort key column exists explicitly in the select list.",4 DM-681,"Parser ignores syntax after LIMIT","Parser stops after the LIMIT condition, believing that it has a complete select statement. It ignores whatever is afterwards.To fix this, we would need to alter the parser to make sure that there isn't garbage afterwards. So we have to make sure that the only thing acceptable afterwards is a semicolon or a comment. The right thing to do is probably to add a grammar rule for this. The easiest thing is to check to see if the entire string was consumed (give-or-take a semicolon), but this would disallow comments.",4 DM-684,"Estimate expected counts of unassociated sources ","We need to have an idea how many sources/forcedSource/diaSources that are not associated with any object we will have to deal with in the database. Can we have a rough estimate? (e.g., per DR).",2 DM-685,"Fine-tune logging messages","Fine-tune log messages in Qserv (what messages are printed, what is the error level, etc)",5 DM-689,"During scons configure : check if mysql isn't runing","Mysqld can't be configured is its running before configuration step.",1 DM-690,"Minor possible enhancements in install procedure","Usefull enhancements : - add swigged target to ""build"" alias (run scons install to see that swigged target are re-builded at install time) Other possibles enhancements : - manage default (i.e. const.py) for server configuration file ? - state.py : where to save state ? print it to sdtout ? ",7 DM-693,"Create tutorial on the use of eups","Create a beginner's guide to the use of EUPS for the DM Developer Guide. Describe the capabilities and basic usage for ""everyday"" user and developer activities. Provide pointers to the EUPS reference manual for advanced usage. ",10 DM-699,"rename git repository qservdata to qserv_testdata","eups package have the same name as their related git repos. Renaming git repos would lead to a more understandable name. Please note that the qserv-testdata may also be cloned from qservdata, and and qservdata be removed. New repos will also have to be distributed with lsst-sw tool.",1 DM-702,"Buildbot CI needs to save manifest file of failed build for later user debug","The manifest file created during a build instance is transient and removed as soon as the next build commences. Due to that volatility, it's important to save the manifest to some well-known location so that the developer responsible for debug and repair can easily setup the failing environment. The location of the manifest file will be provided to the developer(s) in the failure notification.",4 DM-703,"Use of HipChat for Buildbot CI failure notifications should be explored","K-T recommended the use of HipChat rather than email when notifying users of a buildbot build failure. The purpose was twofold: get immediate attention from the developers and help change the culture towards using HipChat more. This Issue is to explore the feasibility of using HipChat for the notifications.",1 DM-704,"Better review notification e-mails","Russell writes: {quote} I think our system for getting code reviewed using JIRA needs some improvements. It seems that people don't always know that they have been assigned to review a ticket. Also, even if I know I have been assigned to review a ticket, I find it hard to find on JIRA. More concretely, I would like to see these improvements: - Much clearer notification that one has been assigned as a reviewer. Presently the email is quite generic and easy to miss. In fact I find that most JIRA notifications are rather hard to read -- it's not always easy to see what has changed and thus why I should care. The signal to noise ratio is poor. - By default a user should see which issues they have been assigned as reviewer when they log into JIRA. (If there is a way to reconfigure the dashboard for this, I'd like to know about it, but it really should be the default). One way to fix this, of course, is to reassig the ticket when putting it into review, but we have good reasons to avoid that. -- Russell {quote} and I added: {quote} In fact, you don't know that the ticket has passed into review unless you scroll all the way to the bottom of the comment. If the comment associated with the change in status is long and you don't scroll all the way down, then you may not know that you were assigned to review. With Trac, the important information was at the top of the e-mail. {quote}",2 DM-705,"CSS keys are too fine-grain, consider merging them together (design)","I've seen a lot of sentiment (from Serge and Daniel) to try and combine keys in CSS. Current design has one key for each table-chunk (leading to tens of thousands of keys in production), and small-grain keys like ""nStripes"", ""nSubStripes"", and ""overlap"" in partitioning are each stored in separate key. This issue will capture discussion, decisions, and implementation of a more compact version. ",10 DM-706,"cleanup extra file names in docstring","Reported by Serge in email: When using doxygen to document C++ source, you can mark a comment block with just: {code} /** @file * Blah blah */ {code} in which case doxygen assumes you want the comment block tied to the file it appears in. We seem to have lots of ""@file ” statements all over the place, which is an extra thing we have to remember to change when renaming files. Is there some reason to do it that way that I’m missing?",1 DM-707,"cleanup exception code in CSS","Reported by Serge: In CssException.h you’ve got: {code} class CssRunTimeException: public std::runtime_error { … }; class CssException_XXXX : public CssRunTimeException { … }; {code} This is inconsistent (shouldn’t it be CssRunTimeException_XXX, or maybe even CssRunTimeError?), lengthy, violates the LSST C++ naming conventions, and doesn’t match the KvInterface docs, which all still talk about a CssException class that does not exist. Can we consider changing this to something more like: {code} class CssError : public std::runtime_error class KeyError : public CssError class NoSuchTable : public KeyError class NoSuchDb : public KeyError class AuthError : public CssError class ConnError : public CssError {code} ? Then we can succinctly throw and catch css::NoSuchTable, css::AuthError etc…",2 DM-708,"IN2P3: Cloud-computing Platform / OpenStack","CC-IN2P3 offer computing resources via Openstack : https://indico.in2p3.fr/getFile.py/access?sessionId=2&resId=0&materialId=0&confId=8236 This platform allow to boot virtual machines on multiples linux distribution. It could be a powerfull tool for continuous integration and testing. Virtual machines are easilly available to all IN2P3 users, only a SSH account on ccage.in2p3.fr is required. ",15 DM-709,"Prepare a fedora64 openstack image which allow to easily build and test Qserv","Qserv packaging procedure requires to often rebuild Qserv and relaunch integration tests. IN2P3 Openstack platform offer next virtual machines : {code:bash} [fjammes@ccage030 ~]$ nova flavor-list {code} | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | | 1 | m1.tiny | 512 | 0 | 0 | | 1 | 1.0 | True | | 15 | cc.windows.small | 4096 | 20 | 0 | | 2 | 1.0 | True | | 16 | cc.windows.xlarge | 8192 | 50 | 0 | | 4 | 1.0 | True | | 2 | m1.small | 2048 | 10 | 20 | | 1 | 1.0 | True | | 3 | m1.medium | 4096 | 10 | 40 | | 2 | 1.0 | True | | 4 | m1.large | 8192 | 10 | 80 | | 4 | 1.0 | True | | 5 | m1.xlarge | 16384 | 10 | 160 | | 8 | 1.0 | True | | 6 | cc.lsst.medium | 4096 | 20 | 40 | | 2 | 1.0 | False | | 7 | cc.lsst.large | 16384 | 20 | 160 | | 8 | 1.0 | False | | 9 | cc.lsst.xlarge | 40000 | 20 | 160 | | 20 | 1.0 | False | cc.lsst.xlarge would allow a quick build/test of new Qserv release.",5 DM-710,"Reduce and comment client configuration file","Client configuration file '~/.lsst/qserv.conf) is used by integration test procedure. Next improvments are required : 1. use templates in it 2. client config file should retrieve templated values from mother config file""",1 DM-720,"Upgrade various external packages","I note that the LSST and HSC versions of external packages are slightly out of sync. I propose uprevving the LSST packages to match as HSC has tested these versions. {quote} cfitsio 3.360 HSC cfitsio 3310+2 sims Winter2014 current b4 b5 b6 b3 doxygen 1.8.2+2 sims b4 Winter2014 current b5 b6 b3 doxygen 1.8.5 HSC eigen 3.1.1+2 Winter2014 current b5 b6 b3 b4 eigen 3.2 HSC fftw 3.3.2+2 Winter2014 current b5 b6 b3 b4 fftw 3.3.3 HSC gsl 1.15+2 Winter2014 current b5 b6 b3 b4 gsl 1.16 HSC minuit2 5.22.00+2 Winter2014 current b5 b6 b3 b4 minuit2 5.28.00 HSC mysqlclient 5.1.65+3 Winter2014 current b5 b6 b3 b4 mysqlclient 5.1.73 HSC pyfits 3.1.2+2 sims b4 Winter2014 current b5 b6 b3 pyfits 3.2 HSC scons 2.1.0+7 sims b4 Winter2014 current b5 b6 b3 scons 2.3.0 HSC sqlite 3.7.14+2 Winter2014 current b5 b6 b3 b4 sqlite 3.8.2 HSC wcslib 4.14 wcslib 4.14+3 b4 Winter2014 current b5 b6 b3 xpa 2.1.14+2 Winter2014 current b5 b6 b3 b4 xpa 2.1.15 HSC {quote} ",20 DM-736,"Rewrite secondary index system and merge empty chunks functionality","We'd like to rewrite the indexing system so that it doesn't depend on a narrow innodb table. We'd also like to fold in the emptychunks functionality. secondary index: dirColumn->(chunkId, subChunkId) emptychunks: hasChunk = chunkID -> bool danielw has a dirt simple implementation of the ""create"" portion of the secondary index.",30 DM-737,"Rendering an IR node tree should produce properly parenthesized output","It appears that rendering a tree of IR nodes doesn't always result in correct generation of parentheses. Consider the following tree: {panel} * OrTerm ** BoolFactor *** NullPredicate **** ValueExpr ***** ValueFactor: ColumnRef(""refObjectId"") ** BoolFactor *** CompPredicate **** ValueExpr ***** ValueFactor: ColumnRef(""flags"") **** Token(""<>"") **** ValueExpr ***** ValueFactor: Const(""2"") {panel} which corresponds to the SQL for: ""refObjectId IS NULL OR flags<>2"". If one prepends this (via {{WhereClause.prependAndTerm()}}) to the {{WhereClause}} obtained by parsing ""... WHERE foo!=bar AND baz<3.14159;"" and renders the result using {{QueryTemplate}}, one obtains: {{... WHERE refObjectId IS NULL OR flags<>2 AND foo!=bar AND baz<3.14159}} This is equivalent to {{... WHERE refObjectId IS NULL OR (flags<>2 AND foo!=bar AND baz<3.14159)}} which doesn't match the parse tree - one should obtain: {{... WHERE (refObjectId IS NULL OR flags<>2) AND foo!=bar AND baz<3.14159}} This issue involves surveying all IR node classes and making sure that they render parentheses properly. {color:gray}(One way we might test for this is to parse queries containing parenthesized expressions where removal of the parentheses changes the meaning of the query. This would give us some IR that we can render to a string and reparse back into IR. If the rendering logic is correct, one should obtain identical IR trees).{color} Other possibilities that might explain the behavior above is that the input tree is somehow invalid or that {{WhereClause.prependAndTerm}} creates invalid IR.",8 DM-740,"Implement abstract base class for approximated or interpolated fields","The user of an Approximate or Interpolate object doesn't care which of these they have, once the object has been constructed, and we should make these inherit from a common base class that only contains an interface for accessing the interpolated/approximated function while making no assumptions about its functional form. The new class will represent a scalar field defined over an integer bounding box, and will have methods for evaluating the field at a point and creating an image of the field. We could also consider giving it arithmetic interoperability with Image. I don't have a strong candidate for the name; I've called it ""BoundedField"" on the HSC side but would like that to be revisited. Adding derived classes to replace the functionality currently in the Approximate and Interpolate classes will be on a separate issue. The main work on this issue is tweaking the design on the HSC side and getting signoff on the final design from LSST developers; the actual coding should be minimal, as it's just an interface and we already have a good starting point for it. The HSC-side issue (which also includes work that is part of DM-1124) is here: https://hsc-jira.astro.princeton.edu/jira/browse/HSC-796 and the associated git commits are here: https://github.com/HyperSuprime-Cam/afw/compare/releases/S14A_0...tickets/DM-796",10 DM-742,"Use geom eups package for installing geometry","Use geom eups package instead of downloading geometry.py during Qserv configuration step.",3 DM-744,"Qserv release (12.04) - final build, testing and cutting release",NULL,4 DM-746,"Simplify (script) install procedure","Install procedure described in READMEs.txt is complex and error prone. It could be encapsulated in two scripts : 1. the first for installing Qserv current version in the eups stack, following the LSST official install procedure, 2. the second, developer-oriented, for installing Qserv from a git repository to the eups stack installed during 1. This two scripts logs could be colorized for better ergonomy. ",3 DM-750,"Integrate sciSQL in eups","In order to become compliant with eups, sciSQL install process may have to be refactored : - provide a scons build and install target which build and install sciSQL binary (.i.e. reduce waf tool features) - provide a configuration procedure (scisql-deploy.py) which install UDF in MySQL datadir, and deploy .so file in MySQL plugin directory. Then it has to be packaged in eups format.",9 DM-751,"Replacing boost system lib with eups libs breaks scons build","While detecting boost, Qserv build system checks for both system lib and then eups lib. This procedure use next code : {code:python} class BoostChecker: def __init__(self, env): self.env = env self.suffix = None self.suffixes = [""-gcc41-mt"", ""-gcc34-mt"", ""-mt"", """"] self.cache = {} pass def getLibName(self, libName): if libName in self.cache: return self.cache[libName] r = self._getLibName(libName) self.cache[libName] = r return r def _getLibName(self, libName): state.log.debug(""BoostChecker._getLibName() LIBPATH : %s, CPPPATH : %s"" % (self.env[""LIBPATH""], self.env[""CPPPATH""])) if self.suffix == None: conf = self.env.Configure() def checkSuffix(sfx): return conf.CheckLib(libName + sfx, language=""C++"", autoadd=0) {code} and this last line run next gcc command : {code:bash} g++ -o .sconf_temp/conftest_10.o -c -g -pedantic -Wall -Wno-long-long -D_FILE_OFFSET_BITS=64 -fPIC -I/data/fjammes/stack/Linux64/protobuf/master-g832d498170/include -I/data/fjammes/stack/Linux64/boost/1.55.0.1/include -I/data/fjammes/stack/Linux64/zookeeper/master-gc48457902f/c-binding/include -I/data/fjammes/stack/Linux64/mysql/master-g5d79af2a50/include -I/data/fjammes/stack/Linux64/antlr/master-gc05368a54f/include -I/data/fjammes/stack/Linux64/xrootd/master-gfc9bfb2059/include/xrootd -Ibuild -I/data/fjammes/stack/Linux64/anaconda/1.8.0/include/python2.7 .sconf_temp/conftest_10.cpp g++ -o .sconf_temp/conftest_10 .sconf_temp/conftest_10.o -L/data/fjammes/stack/Linux64/xrootd/master-gfc9bfb2059/lib -L/data/fjammes/stack/Linux64/protobuf/master-g832d498170/lib -L/data/fjammes/stack/Linux64/antlr/master-gc05368a54f/lib -L/data/fjammes/stack/Linux64/zookeeper/master-gc48457902f/c-binding/lib -L/data/fjammes/stack/Linux64/mysql/master-g5d79af2a50/lib -L/data/fjammes/stack/Linux64/boost/1.55.0.1/lib -lboost_regex-mt scons: Configure: yes {code} As the ""-mt"" suffix is searched before the empty suffix, previous command succeed.In my example boost_regex-mt is a system lib. When launching ""scons build"", then CheckLib only looks for boost in /data/fjammes/stack/Linux64/boost/1.55.0.1/lib, not in /usr/lib/. This behaviour is eups-correct, but prevents to find boost_regex-mt. In this example, a trivial solution is to reverse self.suffixes in python code, but a better solution would be to prevent g++ to use default search paths (e.g. : /usr/lib and /usr/include) in the second command. Is it possible to to it with scons ? Mario, did you meet the same problem with sconsUtils ? Thanks Fabrice",3 DM-754,"Update obs_decam for new CameraGeom","The obs_decam package worked on by Paul and Andy B. needs to be updated to reflect changes in the camera geometry.",6 DM-758,"Stretch: Data and test script specification for daily/automated QA/integration tests","[I have renamed this Epic to better capture the current planned scope] Per discussions I am pulling this Epic into 02C.01.02 [""my"" WBS] The intent for it is to cover the science and scientific programming portion activities associated with setting up a daily automated QA/integration run. The infrastructure activities to enable this are covered by other Epics. It is a stretch because it will be primarily worked on by the Tucson Scientist, still to be identified ICW Tucson Scientific Programmer, still to be identified. JK: In PMCS this would be New Hire LS6",70 DM-760,"Identify Data Set for MiniProduction",NULL,3 DM-761,"Generate data for MiniProduction","Assuming that the data needed for mini-production are to be simulated, the input files need to be created and simulated. The input data then need to be put in a repo with appropriate calibrations.",5 DM-762,"Create scripts to run MiniProduction","Write command line script to run mini production. The scripts should be able to be handed directly to the orca layer.",3 DM-763,"Eliminate local pixel indexing; always use parent instead","As Jim Bosch notes in : Our image classes currently handle two different pixel coordinate systems which differ only by the offset commonly referred to as ""xy0"". The PARENT coordinate system puts the first pixel in the image at xy0; the LOCAL coordinate system always puts the first pixel at (0,0). Our general rule has long been ""always pay attention to xy0"", which implies PARENT, but the Image class itself doesn't: in many operations, including subimage accessors and bbox getters, LOCAL is the default, whereas in others - particularly the pixel and iterator accessors - there isn't even an option to use PARENT. IMO this is the main source of image-offset bugs in the code. We plan to address this issue as follows: * Modify the methods on image-like objects that return pixel iterators or locators based x and/or y index, so that they use parent coordinates. * Eliminate the ImageOrigin enum argument, which appears in image-like constructors, image-like getBBox methods. * Eliminate the image origin string argument in the butler's get and put methods for image-like objects. At the end of this transition all indices will be PARENT indices; the concept of LOCAL will be gone. This will reduce ambiguity and eliminate an argument that is a source of confusion. Furthermore, at Jim's suggestion and K-T's agreement, we plan to do this in two basic stages: * Rename methods or otherwise make changes that should break any code using LOCAL indices. Fix the code accordingly. * Rename methods back and eliminate the ImageOrigin argument. Fix the code accordingly.",53 DM-764,"Exception naming convention","The naming convention for exceptions in pex_exceptions is quite redundant. This issue will make the convention more compact and update all packages that make use of pex_exceptions.",5 DM-765,"Evaluate moving to C++11 for .cc files","Check that {{C\+\+11}} works on .cc files. Make {{C\+\+11}} the default in SconsUtils.",5 DM-766,"Improve afw::CameraGeom::utils code","Some of the utility code in CameraGeom was not completely ported in W13 and documentation is in need of updating.",3 DM-767,"Determine scope of XY0 convention update","It's unclear exactly how much effort will be involved in making a change to how the XY0 is used. If the parent/child argument is removed completely this change could be quite invasive and wide reaching.",2 DM-768,"Does SWIG 3.x work with DM stack?","This task is to evaluate whether we can use SWIG 3.x (and is it stable enough). Assuming SWIG 3.x can be used with the stack, evaluate how much work will be involved in moving over to use it by default.",10 DM-769,"Create scripts to assess and report MiniProduction quality.","Write a command line script to assess the quality of the mini production run. This will involve comparing output data to data produced using a standard stack. The script will provide a report.",7 DM-770,"Create script to clean up after a MiniProduction run","The run of a mini production will produce an output repository. It's likely that we will not want to save all output data. A script to clean up and potentially save parts of the repo is needed.",3 DM-772,"Package log4cxx","Fabrice, can you package log4cxx? I should have asked you earlier, sorry I waited so long, not it becoming urgent! Bill is almost done with his logging prototype and will be turning it into a real package, and we need to have log4cxx packages. Many thanks. log4cxx version 0.10.0, which was released in 4/3/2008 but is still undergoing ""incubation"" at Apache. ",2 DM-774,"XLDB in Rio","Prepare for and attend XLDB-South America in Rio.",15 DM-775,"XLDB-2015 report","Writing the report, most work done by Daniel, with input from Jacek and K-T.",8 DM-776,"Researching partnership opportunities",NULL,8 DM-778,"Restructure and package logging prototype","Restructure and package log4cxx-based prototype (currently in branch u/bchick/protolog). It should go into package called ""log""",8 DM-780,"Access patterns for data store that supports data distribution ","Data distribution related data store includes things like. chunk --> node mapping, locations of chunk replicas, runtime information about nodes (and maybe also node configuration?). Need to understand access patterns - who needs to access, how frequently etc. ",5 DM-781,"research mysql cluster ndb","Checkout mysql cluster ndb from the perspective of data distribution - could it be potentially useful to store data related to data distribution?",2 DM-782,"Automated test should optionally ignore column headers","Some types of queries (like COUNT(*)) may return different column headers in qserv and mysql. This differences break our automated tests which dump and compare complete result including headers. It looks like we will not be able to guarantee that qserv can be made to return the same column headers as mysql except for providing aliases in the query itself. Running those queries without aliases is a legitimate use case so it would be nice to have an option in the test runner which ignores headers for some queries.",4 DM-783,"Disable failing test cases in automated tests","There are currently 4 test cases failing in out automated tests. Until we have a fix we want to disable them.",1 DM-786,"JOIN queries are broken","Running a simple query that does a join: {code} SELECT s.ra, s.decl, o.raRange, o.declRange FROM Object o JOIN Source s USING (objectId) WHERE o.objectId = 390034570102582 AND o.latestObsTime = s.taiMidPoint; {code} results in czar crashing with: {code} 2terminate called after throwing an instance of 'std::logic_error' what(): Attempted subchunk spec list without subchunks. {code} This query has been taken from integration tests (case01, 0003_selectMetadataForOneGalaxy.sql) ",3 DM-787,"Understand DCR amplitudes using realistic distribution of stars",NULL,4 DM-788,"Determine refraction amplitudes as a function of SED",NULL,2 DM-789,"Model refraction amplitudes as a function of SED",NULL,2 DM-790,"Determine DCR ampltiudes as a function of SED",NULL,2 DM-791,"Model DCR amplitudes as a function of SED",NULL,2 DM-792,"Determine requirements on atmospheric measurements to predict refraction/DCR",NULL,2 DM-793,"Put together requirements to model refraction/DCR for a given source",NULL,8 DM-794,"SQL injection in czar/proxy.py","Running automated tests for some queries I observe python exceptions in czar log which look like this: {code} 20140529 19:47:19.364371 0x7faacc003550 INF Query dispatch (7) toUnhandled exception in thread started by Traceback (most recent call last): File ""/usr/local/home/salnikov/qserv-master/build/dist/lib/python/lsst/qserv/czar/proxy.py"", line 78, in waitAndUnlock lock.unlock() File ""/usr/local/home/salnikov/qserv-master/build/dist/lib/python/lsst/qserv/czar/proxy.py"", line 65, in unlock self._saveQueryMessages() File ""/usr/local/home/salnikov/qserv-master/build/dist/lib/python/lsst/qserv/czar/proxy.py"", line 87, in _saveQueryMessages self.db.applySql(Lock.writeTmpl % (self._tableName, chunkId, code, msg, timestamp)) File ""/usr/local/home/salnikov/qserv-master/build/dist/lib/python/lsst/qserv/czar/db.py"", line 95, in applySql c.execute(sql) File ""/u2/salnikov/STACK/Linux64/mysqlpython/1.2.3+8/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/cursors.py"", line 174, in execute self.errorhandler(self, exc, value) File ""/u2/salnikov/STACK/Linux64/mysqlpython/1.2.3+8/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/connections.py"", line 36, in defaulterrorhandler raise errorclass, errorvalue _mysql_exceptions.ProgrammingError: (1064, ""You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'r' AND sce.tract=0 AND sce.patch='159,3';', 1401410839.000000)' at line 1"") ok 0.000532 seconds {code} I believe this is due to how query string is being constructed in czar/proxy.py: {code:py} class Lock: writeTmpl = ""INSERT INTO %s VALUES (%d, %d, '%s', %f);"" # ................... self.db.applySql(Lock.writeTmpl % (self._tableName, chunkId, code, msg, timestamp)) {code} If {{msg}} happens to contain quotes then resulting query is broken. One should not use Python formatting to construct query strings, instead the parameters should be passed directly to {{cursor.execute()}} method. ",2 DM-800,"Zookeeper times out","I noticed running some queries, leaving system up and them returning few hours later and running more queries can result in: {code} ZOO_ERROR@handle_socket_error_msg@1723: Socket [127.0.0.1:12181] zk retcode=-4, errno=112(Host is down): failed while receiving a server response {code} It needs to be investigated (if we can reproduce) ",3 DM-803,"Provide a ""stackfitCalib"" for coadds","Coadds have a Psf class that returns the proper sum of input PSFs at a point. We need the same functionality for the Calib object associated with the Coadd Exposure. This will require making a Calib a baseclass (it currently doesn't have a virtual dtor (although it does have virtual protected members)) ",4 DM-813,"PhotoCalTask doesn't return information about which stars were used in calibration","The PhotoCalTask returns numpy arrays of the source and reference fluxes (and errors) of matched ""good"" photometric objects, typically stars. However, while estimating the zero point, it clips outliers so the actual list of objects used is shorter. Please add another output to the returned struct, a numpy Bool array ""good"", to indicate which objects are kept. ",1 DM-814,"Cleanup in core/examples and core/doc","- core/examples and core/doc seems to be out of data. Some cleanup here would be welcome.",1 DM-815,"Define filter for Epics and mapping of Epics to WBS",NULL,2 DM-817,"qserv have to use boost from stack","To quote Jacek and KT: {code} Andy, re dm-751, KT says never use the system version. J. {code} So we need to switch qserv to eups-boost. This should be easy once DM-751 is done, just add boost to qserv.table. Then one can remove conditional part of {{BoostChecker}} which works with system-installed boost. ",1 DM-820,"Simplify copying tables while adding columns","Currently, if I want to copy a table while adding a few columns (as specified by schema in the example) I need to do something like: {code} cat = afwTable.SourceCatalog(schema) cat.table.defineCentroid(srcCat.table.getCentroidDefinition()) cat.table.definePsfFlux(srcCat.table.getPsfFluxDefinition()) # etc. scm = afwTable.SchemaMapper(srcCat.getSchema(), schema) for schEl in srcCat.getSchema(): scm.addMapping(schEl.getKey(), True) cat.extend(srcCat, True, scm) {code} Please make this easier! For example - by adding a flag to the SchemaMapper constructor that automatically does the addMapping (should this be the default?) - by making it possible to copy all the slots (maybe this'll be the case when the new alias scheme is implemented?). Maybe we just need a new method: {code} cat = srcCat.extend(schema) {code} that does all the above steps.",4 DM-827,"Reimplement C++/Python Exception Translation","I'd like to reimplement our Swig bindings for C++ exceptions to replace the ""LsstCppException"" class with a more user-friendly mechanism. We'd have a Python exception hierarchy that mirrors the C++ hierarchy (generated automatically with the help of a few Swig macros). These wrapped exceptions could be thrown in Python as if they were pure-Python exceptions, and could be caught in Python in the same language regardless of where they were thrown. We're doing this as part of a ""Measurement"" sprint because we'd like to define custom exceptions for different kinds of common measurement errors, and we want to be able to raise those exceptions in either language.",8 DM-828,"Design Prototypes for C++ Algorithm API","We've never really been happy with the new design for the C++ algorithm API, and Perry and Jim have a few ideas to fix this that need to be fleshed out. Each of the subtasks of this issue will correspond to a different design idea. Ideally, for each one, we'll try to do a nearly-complete conversion of the SdssShape algorithm (as a good example of a complicated algorithm) to see how these ideas work in practice.",6 DM-829,"Algorithm API without (or with optional) Result objects","In this design prototype, I'll see how much simpler things could be made by making the main algorithm interface one that sets record values directly, instead of going through an intermediate Result object. Ideally the Result objects would still be an option, but they may not be standardized or reusable.",3 DM-832,"add persistable class for aperture corrections","We need to create a persistable, map-like container class to hold aperture corrections, with each element of the container being an instance of the class to be added in DM-740. A prototype has been developed on DM-797 on the HSC side: https://hsc-jira.astro.princeton.edu/jira/browse/HSC-797 and the corresponding code can be found on these changesets: https://github.com/HyperSuprime-Cam/afw/compare/32d7a8e7b75da6f5327fee65515ee59a5b09f6c7...tickets/DM-797",2 DM-833,"implement coaddition for aperture corrections","We need to be able to coadd aperture corrections in much the same way we coadd PSFs. See the HSC-side HSC-798 and HSC-897 implementation for a prototype: https://hsc-jira.astro.princeton.edu/jira/browse/HSC-798 https://hsc-jira.astro.princeton.edu/jira/browse/HSC-897 with code here: https://github.com/HyperSuprime-Cam/meas_algorithms/compare/d2782da175c...u/jbosch/DM-798 https://github.com/HyperSuprime-Cam/meas_algorithms/compare/c4fcab3251...u/price/HSC-897a https://github.com/HyperSuprime-Cam/pipe_tasks/compare/6eb48e90be12d...u/price/HSC-897a",3 DM-834,"reduce code- and object-duplication in aperture correction and PSF coaddition","The aperture correction code to be added on DM-833 will likely not be as closely integrated with CoaddPsf as it could be, because the original design of CoaddPsf didn't anticipate the addition of other, similar classes. We should work on allowing these classes to share code.",8 DM-836,"design Array fields for table version 1","While we're trying to eliminate the need for compound fields in afw::table, Arrays present a few problems. We could use FunctorKeys, the way we plan to use other compound fields, but here we need to guarantee that the per-element keys are contiguous, and we also be able to support views. We also need to determine the naming scheme. Finally, we need to make sure these work with aliases and slots.",6 DM-837,"Rewrite multiple-aperture photometry class","We've never figured out how to handle wrapping multiple-aperture photometry algorithms. They can't use the existing Result objects - at least not out of the box. We should try to write a new multiple-aperture photometry algorithm from the ground up, using the old ones on the HSC branch as a guide, but not trying to transfer the old code over. The new one should: - Have the option of using elliptical apertures (as defined by the shape slot) or circular apertures. - Have a transition radius at which we switch from the sinc photometry algorithm to the naive algorithm (for performance reasons).",2 DM-839,"Rename methods that return pixel iterators and locators in image-like classes, and change to use parent indexing","Image-like classes have a large number of methods that return pixel iterators and locators based on row and/or column. We wish to change these to use parent indexing (making row and column relative to XY0). As the first step in doing this, rename all these methods and modify them to parent indexing. Renaming will help identify and fix all code that uses these methods.",6 DM-840,"Change code so ImageOrigin must be specified (temporary)","Image-like classes have a getBBox method and various constructors that use an ImageOrigin argument which in most or all cases defaults to LOCAL. As the first stage in cleaning this up, try to break code that uses the default as follows: * Remove the default from getBBox(ImageOrigin) so an origin must be specified. * Change the default origin of constructors to a temporary new value UNDEFINED * Modify code that uses image origin to fail if origin is needed (it is ignored if bbox is empty) and is UNDEFINED. Note: this is less safe than changing constructors to not have a default value for origin, because the error will be caught at runtime rather than compile time. However, that is messy because then the bounding box will also have to be always specified, and possibly an HDU, so it would be a much more intrusive change.",2 DM-841,"Change data butler I/O of image-like objects to require imageOrigin if bbox specified (temporary)","As part of making PARENT the default for image origin, change the data butler to require that imageOrigin be specified if bbox is specified when reading or writing image-like objects. Note: this ticket turns out to be unnecessary, as all the few necessary change are done as part of DM-840.",2 DM-842,"Update code that uses pixel iterators and accessors","Fix all code that uses pixel iterators and locators as per the changes in DM-839. This affects roughly 60 files distributed over many packages.",16 DM-843,"Restore names of methods that return pixel iterators and locators","Restore the names of methods that return pixel iterators and pixel locators on image-like classes. (This is part of the final stage of eliminating LOCAL pixel indexing).",2 DM-844,"Eliminate ImageOrigin argument","Eliminate the ImageOrigin enum and argument from image-like classes.",2 DM-845,"Eliminate image origin argument from butler for (un)persisting image-like objects","Eliminate the image origin argument for butler get and put when dealing with image-like objects.",2 DM-846,"Change code to always specify image origin (temporary)","The next step after DM-840 is to change all code that uses image-like getBBox and constructors that take an image origin argument to always specify the origin (DM-840 will break any code that tries to use a default image origin). This results in a working stack after DM-840. Neither this nor DM-840 will be merged to master, though it can be used by HSC and other users as they transition to a default image origin of PARENT. See DM-1176 for the final step.",10 DM-847,"Update code to use restored names for methods that return pixel iterators and locators","For all code changed in DM-842, change it again to use the restored names for methods that return pixel iterators and locators. This task is much simpler than DM-842 because it is merely renaming methods. Nonetheless, it touches many files in many packages.",4 DM-848,"Specify ImageOrigin = PARENT in all cases","For all code that uses the ImageOrigin argument (either explicitly or using a bbox and the default value) change the code to use an explicit ImageOrigin of PARENT. In other words, update all code to match the changes in DM-840 and DM-841. This affects 40-ish modules in many packages. I am concerned about testing the modified code because we are missing unit tests for so many tasks. But we can do some data processing and check those results. I will want some help with that.",12 DM-849,"Eliminate use of ImageOrigin argument","For all code that uses the ImageOrigin argument, eliminate the use of that argument. In other words, update all code to match the changes in DM-844 and DM-845, This affects all the code affected by DM-848, plus any code that was already using ImageOrigin=PARENT, but is a simpler change.",5 DM-854,"duplicate column name when running near neighbor query","Running a simplified version of near neighbor query on test data from case01: {code} SELECT DISTINCT o1.objectId, o2.objectId FROM Object o1, Object o2 WHERE scisql_angSep(o1.ra_PS, o1.decl_PS, o2.ra_PS, o2.decl_PS) < 1 AND o1.objectId <> o2.objectId {code} Result in an error on the worker: {code} Foreman:Broken! ,q_38f9QueryExec---Duplicate column name 'objectId' Unable to execute query: CREATE TABLE r_13237cd4cfc9e0fa01497bcf\ 67a91add2_6630_0 SELECT o1.objectId,o2.objectId FROM Subchunks_LSST_6630.Object_6630_0 AS o1,Subchunks_LSST_6630.Object_6630_0 AS o2\ WHERE scisql_angSep(o1.ra_PS,o1.decl_PS,o2.ra_PS,o2.decl_PS)<1 AND o1.objectId<>o2.objectId; {code} It is fairly obvious what is going on. ""SELECT t1.x, t2.x"" is perfectly valid, but if we add ""INSERT INTO SELECT t1.x, t2.x"", we need to add names, eg. something like ""INSERT INTO SELECT t1.x as x1, t2.x as x2""",8 DM-860,"Passing error messages from czar to end user - design",NULL,3 DM-861,"Provide a detailed integration tests report","Test output is very low level. Indeed only a verbose logfile and SLQ queries output are currently available. Furthermore failing queries (i.e. .sql.FIXME) aren't launched. This ouput could be leveraged to a detailed html web report (using sbadmin for example : http://startbootstrap.com/templates/sb-admin/) which could : - launch all queries, even failing ones, - print Qserv services status after each query, - execution time for each query, - information about MySQL and Qserv results differences, - href to Qserv services log files - all other interesting information about queries execution ",10 DM-862,"apr and apt_util packages do not install shared library","When we installed apr and apr_utils packages (as a dependency of new log4cxx package, see DM-772) we discovered that both these packages only build static libraries but no shared libs are installed. This is problematic if mixed with shared libs and we use shared libs everywhere else. We certainly need to build shared libs for these packages, this ticket is to follow up on this problem.",2 DM-863,"near neighbor does not return results","A query from qserv_testdata (case01/queries/1051_nn.sql) runs through Qserv, but it returns no results, while the same query run on myql does return results. The exact query for qserv is: {code} SELECT o1.objectId AS objId FROM Object o1, Object o2 WHERE qserv_areaspec_box(0, 0, 0.2, 1) AND scisql_angSep(o1.ra_PS, o1.decl_PS, o2.ra_PS, o2.decl_PS) < 1 AND o1.objectId <> o2.objectId; {code}",1 DM-869,"disable extraneous warnings from boost (gcc 4.8)","Compiling qserv on ubuntu 14.04 (comes with gcc 4.8.2) results in huge number of warnings coming from boost. We should use the flag ""-Wno-unused-local-typedefs"".",1 DM-870,"XLDB Rio - workshop report",NULL,3 DM-873,"XLDB - strategic positioning","Discussions with strategic partners. Improving website and adding new context (community, speakers). 1-pager document",3 DM-874,"W'14 newinstall.sh picks up wrong python?","newinstall.sh fails with: Installing the basic environment ... Traceback (most recent call last): File ""/tmp/test_lsst/eups/bin/eups_impl.py"", line 11, in ? import eups.cmd File ""/tmp/test_lsst/eups/python/eups/__init__.py"", line 5, in ? from cmd import commandCallbacks File ""/tmp/test_lsst/eups/python/eups/cmd.py"", line 38, in ? import distrib File ""/tmp/test_lsst/eups/python/eups/distrib/__init__.py"", line 30, in ? from Repositories import Repositories File ""/tmp/test_lsst/eups/python/eups/distrib/Repositories.py"", line 8, in ? import server File ""/tmp/test_lsst/eups/python/eups/distrib/server.py"", line 1498 mapping = self._noReinstall if outVersion and outVersion.lower() == ""noreinstall"" else self._mapping ^ SyntaxError: invalid syntax Perhaps from running the wrong version of python. Full script/log is attached. ",1 DM-875,lsst_dm_stack_demo,"lsst-dm_stack_demo has obsolete benchmark files (circa Release 7.0) which fail to serve the purpose of validating, for the user, the correct functioning of a freshly built Release v8.0 stack. At the very least, the benchmark files should be regenerated for each official Release. Tasks: (1) Build the benchmark files for Release v8.0 (2) Debate (a) recommending the use of 'numdiff' to check if the output is within realistic bounds. Or, (b) develop another procedure to better show how the current algorithms compare to the algorithms used at the benchmarked Release. (3) Depending on result of the debate on #2: for: (a) provide appropriate 'numdiff' command invocation in manual.; for (b) implement the new procedure.",40 DM-882,"Segmentation fault from writing dotted FITS header keywords","Observed on HSC, but I'm not aware of any fix for this on the LSST side either. {code:sh} pprice@tiger3:~ $ gdb python GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1) (gdb) r Starting program: /tigress/HSC/products-20130212/Linux64/python/2.7.6/bin/python >>> from lsst.afw.image import ExposureF >>> exp = ExposureF(1,1) >>> exp.getMetadata().add(""A.B.C.D"", 12345) >>> exp.writeFits(""test.fits"") Program received signal SIGSEGV, Segmentation fault. lsst::daf::base::PropertySet::combine (this=0x1eda370, source=...) at src/PropertySet.cc:746 746 if (dj == _map.end()) { (gdb) w #0 lsst::daf::base::PropertySet::combine (this=0x1eda370, source=...) at src/PropertySet.cc:746 #1 0x00002aaabac068dc in lsst::daf::base::PropertyList::deepCopy (this=0x1ddd4c0) at src/PropertyList.cc:74 #2 0x00002aaab7ba559e in lsst::afw::image::MaskedImage::writeFits (this=0x1f1d8f0, fitsfile=..., metadata=, imageMetadata=..., maskMetadata=..., varianceMetadata=...) at src/image/MaskedImage.cc:569 #3 0x00002aaab7b0a7f0 in lsst::afw::image::Exposure::writeFits (this=0x1f1d8d0, fitsfile=...) at src/image/Exposure.cc:251 #4 0x00002aaab7b0a9fc in lsst::afw::image::Exposure::writeFits (this=0x1f1d8d0, fileName=""test.fits"") at src/image/Exposure.cc:239 #5 0x00002aaab6b91614 in _wrap_ExposureF_writeFits__SWIG_0 (self=, args=) at python/lsst/afw/image/imageLib_wrap.cc:160786 #6 _wrap_ExposureF_writeFits (self=, args=) at python/lsst/afw/image/imageLib_wrap.cc:29886 … {code} This appears to only affect dotted keywords (which should be supported through use of {{HIERARCH}}, and even if they're not supported, this should never fail with a segfault).",4 DM-887,"Optimize template engine used in configuration tool","string.Template and string interpolation are quite weak, some additional feature would be welcomed : - interpolation interprets wrongly ""%d"" in template files as template - string.template doesn't manage correctly non referenced template parameters present in template files.",4 DM-895,"Use an existing qserv_run_dir with a new Qserv instance/binary","Here's what should be added to qserv-configure.py : - edit $QSERV_RUN_DIR/admin/qserv.conf and change Qserv instance dir to current one (which qserv-configure.sh), - check compliance of QSERV_RUN_DIR with new Qserv instance (version check ?) and/or update configuration files, - re-initialize services, if needed, without breaking already loaded data. ",4 DM-898,"unset BASH_ENV in newinstall.sh or surrounding instructions","Nutbar users such as myself may have BASH_ENV set, which can mess with your carefully constructed shell environments. Unsetting BASH_ENV should help. This could perhaps go in newinstall.sh, or the documentation alongside where we suggest they unset other variables: https://confluence.lsstcorp.org/display/LSWUG/Building+the+v8.0+LSST+Stack+from+Source",1 DM-900,"Update PhoSim tutorial to use CatSim for creating instance catalogs","Update the Process PhoSim Images tutorial (https://confluence.lsstcorp.org/display/LSWUG/Process+PhoSim+Images) to use CatSim to generate instance catalogs, once CatSim documentation is available. This will obviate the need for the helper script refCalCat.py. ",2 DM-903,"SourceDetectionTask should only add flags.negative if config.thresholdParity == ""both""","The SourceDetectionTask always adds ""flags.negative"" to the schema (if provided) but it is only used if config.thresholdParity == ""both"". As adding a field to a schema requires that the table passed to the run method have that field this is a significant nuisance when reusing the task. Please change the code to only modify the schema if it's going to set it. ",1 DM-911,"Provide Task documentation for DipoleMeasurementTask","See Summary. ",2 DM-912,"Provide Task documentation for PsfMatchTask","See Description (it's currently called PsfMatch) ",4 DM-913,"Provide Task documentation for ImagePsfMatchTask","See summary",2 DM-914,"Provide Task documentation for SnapPsfMatchTask","See summary",2 DM-915,"Provide Task documentation for AssembleCcdTask","See summary",4 DM-916,"Provide Task documentation for IsrTask","See Summary",4 DM-926,"Provide Task documentation for SourceDeblendTask","See Summary",2 DM-927,"Provide Task documentation for CmdLineTask","See Summary",4 DM-928,"Provide Task documentation for RepairTask","See Summary",4 DM-929,"How to write your own command line task, including how-to-retarget sub-tasks","Please provide documentation on how to write a command line task, and how to retarget tasks (e.g. reusing bits of IsrTask for a new camera) The documentation should include a complete annotated example. ",4 DM-930,"Improve install/configuration/tests documentation and migrate it to reST format","This ticket propose to migrate README and README-devel to reST format (see http://sphinx-doc.org/rest.html). The output is located here : http://lsst-web.ncsa.illinois.edu/~fjammes/qserv-doc/ Furthermore this ticket wil integrate Andy S. DM-622 value-added remarks about Qserv embedded documentation. {quote} README.txt needs a bit of formatting, whole ""NOTE FOR DEVELOPERS"" is one long line which may need scrolling depending on what do you use to read the file, same applies to README-devel.txt The install procedure in README.txt implies that the whole stack has to be installed including eups. If people have some part of it installed already the it would probably be better to reuse existing stack. Shall we spit install instructions into ""Install eups (if not installed already)"" and ""Install qserv""? README-devel.txt says ""Once Qserv is installed..."", I don't think that we need or want to install whole qserv before we start development (what if qserv is not available yet for the platform I'm trying to test). What probably needed is installed dependencies, and this should be covered by the comments before 'setup -r .' {quote} ",1 DM-933,"Photometric calibration uses a column ""flux"" not the specified filter unless a colour term is active","The photometric calibration code uses a field ""flux"" in the reference catalog to impose a magnitude limit. If a colour term is specified, it uses the primary and secondary filters to calculate the reference magnitude, but if there is no colour term it uses the column labelled ""flux"" and ignores the filtername. Please change the code so that ""flux"" is ignored, and the flux associated with filterName is used.",1 DM-936,"Replace getXXXKey for slots with returning functorKeys similar to existing compound Keys","Make any changes in the Functor Key capabilities necessary to support getXXXKey, getXXXErrKey, and getXXXFlagKey for the 3 different types of slots. Change these routines in Source.h.m4 to return FunctorKeys for both version 0 and version 1 tables. Then fixup any compilation breaks on the C++ size which this causes. Remove the version 1 specific accessors. I am assigning this to Jim to confirm that the first part is done, then he can assign the remainder to Perry either by reassigning, or as a subtask.",4 DM-944,"Package xrootd-4.0.0rc3-qsClient2 with eups",NULL,1 DM-945,"Prevent conflict related to non-unique temporary files created during SciSQL install","SciSQL install sometime fails with next message : {code:bash} [1/2] MySqlScript: scripts/install.mysql [2/2] MySqlScript: scripts/demo.mysql Running testHtm : OK Running testSelect : OK Running testAngSep.py : OK Running testMedian.py : OK Running testPercentile.py : OK Running testS2CPoly.py : OK Running testS2PtInBox.py : OK Running testS2PtInCircle.py : OK Running testS2PtInEllipse.py : OK Running docs.py : FAIL [see /usr/local/home/salnikov/qserv-run/u.fjammes.DM-622-g86a30ec72a/tmp/scisql-0.3.2/build/tools/docs.log] 9 tests passed, 1 failed, and 0 failed to run One or more sciSQL unit tests failed make: *** [install] Error 1 -- CRITICAL: Error code returned by command : / u s r / l o c a l / h o m e / s a l n i k o v / q s e r v - r u n / u . f j a m m e s . D M - 6 2 2 - g 8 6 a 3 0 e c 7 2 a / t m p / c o n f i g u r e / s c i s q l . s h $ cat /usr/local/home/salnikov/qserv-run/u.fjammes.DM-622-g86a30ec72a/tmp/scisql-0.3.2/build/tools/docs.log rm: cannot remove `/tmp/scisql_demo_ccds.tsv': Operation not permitted Failed to run documentation example: rm -f /tmp/scisql_demo_ccds.tsv ERROR 1086 (HY000) at line 4: File '/tmp/scisql_demo_ccds.tsv' already exists {code} It looks like test uses non-unique temporary file name and someone already tried to run installation on the same host. ",2 DM-946,"Improve management of qserv-run-dir in start/stop scripts.","qserv-start.sh by default tries again to use /usr/local/home/salnikov/qserv-run/2014_05.0 directory. There may be a confusion if we have multiple run directories. Would it be better to install qserv start/stop scripts into run directory itself so that they don't need to guess anything?",2 DM-951,"Add Doxygen documentation on rebuilds","Master-branch doxygen documentation should be rebuild on every full master build.",20 DM-953,"Move qserv-testdata.py to qserv_testdata package","Andy S. suggests this during DM-622 review, but this needs to be studied deeply, as qserv_testdata goal is to store large datafile, difficult to version in git.",3 DM-954,"Clearer and shorter output for qserv-configure.py","qserv-configure.py produces a lot of output which could be confusing if people try to look at it and understand it. It may be better to reduce it to something that just indicates that each step is completed successfully.",1 DM-956,"Buildbot email should state if the build used master only or included other branches","Buildbot build status report currently doesn't state if the build was only for master or included other branches requested by the user.",4 DM-957,"Use aliases to clean up table version transition","The addition of schema aliases on DM-417 should allow us to clean up some of the transitional code added on DM-545, as we can now alias new versions of fields to the old ones and vice versa.",2 DM-958,"Move table versions to Schema","We've added a version number to afw::table::BaseTable to help with the transition to a new approach with different naming conventions and FunctorKeys instead of compound keys. However, it looks like Schema objects need to know about this version number as well, in order to change Subschema objects to split/join using underscores instead of periods. That means we'll have to move the version down into the schema object, which may affect a lot of downstream code. Other than touching a lot of code, the changes should be trivial.",4 DM-963,"measAlg.interpolateOverDefects doesn't accept a python list of Defects","The python binding of interpolateOverDefects should accept a python list of Defects as well as a swig-wrapped std::vector; it doesn't (try using a python list in test818 in meas_algorithms/tests/Interp.py) ",1 DM-964,"Include aliases in Schema introspection","Schema stringification and iteration should include aliases somehow. Likewise the extract() Python methods.",1 DM-966,"fix int/long conversion on 32-bit systems and selected 64-bit systems","tests/wrap.py fails in pex_config on 32-bit systems and some 64-bit systems (including Ubuntu 14.04) with the following: {code:no-linenum} tests/wrap.py ...EE.E. ====================================================================== ERROR: testDefaults (__main__.NestedWrapTest) Test that C++ Control object defaults are correctly used as defaults for Config objects. ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/wrap.py"", line 89, in testDefaults self.assert_(testLib.checkNestedControl(control, config.a.p, config.a.q, config.b)) File ""/home/boutigny/CFHT/stack_5/build/pex_config/tests/testLib.py"", line 987, in checkNestedControl return _testLib.checkNestedControl(*args) TypeError: in method 'checkNestedControl', argument 2 of type 'double' ====================================================================== ERROR: testInt64 (__main__.NestedWrapTest) Test that we can wrap C++ Control objects with int64 members. ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/wrap.py"", line 95, in testInt64 self.assert_(testLib.checkNestedControl(control, config.a.p, config.a.q, config.b)) File ""/home/boutigny/CFHT/stack_5/build/pex_config/tests/testLib.py"", line 987, in checkNestedControl return _testLib.checkNestedControl(*args) TypeError: in method 'checkNestedControl', argument 2 of type 'double' ====================================================================== ERROR: testReadControl (__main__.NestedWrapTest) Test reading the values from a C++ Control object into a Config object. ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/wrap.py"", line 82, in testReadControl config.readControl(control) File ""/home/boutigny/CFHT/stack_5/build/pex_config/python/lsst/pex/config/wrap.py"", line 212, in readControl __at=__at, __label=__label, __reset=__reset) File ""/home/boutigny/CFHT/stack_5/build/pex_config/python/lsst/pex/config/wrap.py"", line 217, in readControl self.update(__at=__at, __label=__label, **values) File ""/home/boutigny/CFHT/stack_5/build/pex_config/python/lsst/pex/config/config.py"", line 515, in update field.__set__(self, value, at=at, label=label) File ""/home/boutigny/CFHT/stack_5/build/pex_config/python/lsst/pex/config/config.py"", line 310, in __set__ raise FieldValidationError(self, instance, e.message) FieldValidationError: Field 'a.q' failed validation: Value 4 is of incorrect type long. Expected type int For more information read the Field definition at: File ""/home/boutigny/CFHT/stack_5/build/pex_config/python/lsst/pex/config/wrap.py"", line 184, in makeConfigClass fields[k] = FieldCls(doc=doc, dtype=dtype, optional=True) And the Config definition at: File ""/home/boutigny/CFHT/stack_5/build/pex_config/python/lsst/pex/config/wrap.py"", line 131, in makeConfigClass cls = type(name, (base,), {""__doc__"":doc}) ---------------------------------------------------------------------- Ran 8 tests in 0.017s FAILED (errors=3) {code} There is a partial fix on u/jbosch/intwrappers; this seems to work for Ubuntu 14.04, but not on 32-bit systems.",2 DM-967,"qserv-configure.py is broken in master","It looks like there was a bug introduced either during the merge of DM-622 with master or right before that. Running {{qserv-configure.py}} from master fails now: {code} $ qserv-configure.py File ""/usr/local/home/salnikov/qserv-master/build/dist/bin/qserv-configure.py"", line 229 (""Do you want to update user configuration file (currently pointing ^ SyntaxError: EOL while scanning string literal {code} I assign this to myself, Fabrice is on vacation now and we need to fix this quickly.",1 DM-973,"Create a ""stub"" package that checks for system dependencies for Qserv","Create a special ""stab"" package that checks and reports missing dependencies.",2 DM-975,"setup standard aliases for frequently-used measurements","Frequently-used measurement fields such as classification and pixel flags should have shortened aliases that can be used instead of their full, package-qualified versions. In some cases, these may serve as a sort of slot (e.g. we may have multiple classifications algorithms someday). In this issue, we should audit all current measurement algorithm fields that don't already have a slot that works for them and consider whether there should be a standard alias. We also need to work out a system for defining these aliases, probably in the config for SingleFrameMeasurementTask.",4 DM-976,"Detailed documentation for meas_base tasks","We should follow RHL's example for detailed task documentation and document all meas_base tasks.",2 DM-977,"Documentation audit and cleanup for meas_base plugins","Many meas_base Plugins and Algorithms have poor documentation, including several whose documentation is a copy/paste relic from some other algorithm. These need to be fixed.",2 DM-978,"add base class for measurement tasks","We should consider adding a base class for measurement tasks (SingleFrameMeasurementTask, ForcedMeasuremedTask) that includes the callMeasure methods. I'm hoping this will help cleanup callMeasure and improve code reuse.",1 DM-979,"Add FunctorKey to replace Coord compound keys","Unlike previous FunctorKey replacements, Coord compound Keys are used in the minimal schema for SimpleTable and SourceTable, which makes removing them problematic.",1 DM-980,"convert measurement algorithms in ip_diffim","ip_diffim includes a few measurement algorithms which need to be converted to the new framework.",5 DM-981,"convert measurement algorithms in meas_extensions_shapeHSM","This is a low-priority ticket to replace the old-style plugins in meas_extensions_shapeHSM with new ones compatible with meas_base. As this isn't a part of the main-line stack, we should delay it until other the meas_base conversion is nearly (or perhaps fully) complete.",3 DM-982,"convert meas_extensions_photometryKron to new measurement framework","This is a low-priority ticket to replace the old-style plugins in meas_extensions_photometryKron with new ones compatible with meas_base. As this isn't a part of the main-line stack, we should delay it until other the meas_base conversion is nearly (or perhaps fully) complete.",3 DM-983,"Finish the Log packaging","Finish the work started by Bill (DM-778)",6 DM-984,"allow partial measurement results to be set when error flag is set","We need to be able to return values at the same time that an error flag is set. The easiest way to do this is to have Algorithms take a Result object as an output argument rather than return it. We'll revisit this design later. ",2 DM-989,".my.cnf in user HOME directory breaks setup script","Presence of {{.my.cnf}} file in the user HOME directory crashes {{qserv-configure.py}} script if parameters in {{.my.cnf}} conflict with parameters in {{qserv.conf}}. How to reproduce: * create .my.cnf file in the home directory: {code} [client] user = anything # host/port and/or socket host = 127.0.0.1 port = 3306 socket = /tmp/mysql.sock {code} * try to run {{qserv-configure}}, it fails with error: {code} /usr/local/home/salnikov/qserv-run/u.salnikov.DM-595/tmp/configure/mysql.sh: connect: Connection refused /usr/local/home/salnikov/qserv-run/u.salnikov.DM-595/tmp/configure/mysql.sh: line 13: /dev/tcp/127.0.0.1/23306: Connection refused ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) {code} It looks like {{~/.my.cnf}} may be a left-over from some earlier qserv installation. If I remove it and re-run {{qserv-configure.py}} now it's not created anymore. Maybe worth adding some kind of protection to {{qserv-configure.py}} in case other users have this file in their home directory.",2 DM-991,"add query involving a blob to the integration tests","We need to add a query (or more?) to the qserv_testdata that involve blobs. Blobs are interesting because they might break some parts of the qserv if we failed to escape things properly etc. ",2 DM-993,"improve message from qserv_testdata","Currently, when I try to run qserv-benchmark but qserv_testdata was not setup, I am getting {code} CRITICAL Unable to find tests datasets. -- FOR EUPS USERS : Please run : eups distrib install qserv_testdata setup qserv_testdata FOR NON-EUPS USERS : Please fill 'testdata_dir' value in ~/.lsst/qserv.conf with the path of the directory containing tests datasets or use --testdata-dir option. {code} It is important to note in the section for eups users that this has to be called BEFORE qserv is setup, otherwise it has no effect. ",1 DM-994,"make slot config validation more intelligent","Slot config validation currently assumes that field names match plugin names, which is not always a safe assumption. This can prevent certain algorithms from being used in slots. We probably can't do this validation in Config.validate(); we need to check in the measurement Task constructor after the schema has already been constructed.",1 DM-995,"Make NoiseReplacer a context manager","I think the API for NoiseReplacer could be made more idiomatic (and possibly safer) by turning it into a ""context manager"" (i.e. so it can be used with the ""with"" statement).",1 DM-998,"Adapt integration test to multi-node setup v1","Following DM-595 we can start qserv in multi-node configuration. Next step is to be able to run integration tests in that setup. This needs a bit of understanding how to distribute chunks between all workers in a cluster and how to load data in remote mysql server.",10 DM-999,"rename config file(s) in Qserv","Rename local.qserv.cnf to qserv-czar.cnf. It is quite likely there are some other config files that would make sense to rename. If you see some candidates, let's discussion on qserv-l and do the renames.",1 DM-1001,"Modify assertAlmostEqual in ip_diffim subtractExposures.py unit test","In unit test, the comparison self.assertAlmostEqual(skp1[nk][np], skp2[nk][np], 4) fails. However if changed to self.assertTrue(abs(skp1[nk][np]-skp2[nk][np]) < 10**-4) which is the desired test, this succeeds. This ticket will remove all assertAlmostEquals from subtractExposure.py and replace with the fundamental comparison operator of the absolute value of the differences.",1 DM-1002,"Research Apache Mesos and Google Kubernetes","It'd be good to check out Apache Mesos and Google Kubernetes and see how relevant they are for Qserv / LSST Database ",4 DM-1004,"Provide Task documentation for ModelPsfMatchTask","See Description (it's currently called PsfMatch) ",2 DM-1005,"Migrate Qserv worker code to the new logging system",NULL,8 DM-1010,"fix names of meas_base plugins to match new naming standards","Some meas_base plugins still have old-style algorithm names.",1 DM-1011,"Remove use of compound fields in minimal schema","If we want to ultimately remove compound fields, we need to remove the ""coord"" field from the SimpleTable/SourceTable minimal schema, and provide a different way to get ra,dec from a source.",4 DM-1012,"remove temporary workaround in new SkyCoord algorithm","SingleFrameSkyCoordPlugin is using the Footprint Peak, not the centroid slot. According to comments in the code, this is a workaround for some problem with centroids. This needs to be fixed.",1 DM-1013,"Classification should set flags upon failure","The classification algorithm claims it can never fail. It can, and should report this.",2 DM-1014,"Detailed documentation for GaussianCentroid","We need more detailed documentation for the GaussianCentroid algorithm, in terms of how it actually computes the centroid. We (Jim and Perry) have done what we can, but we need help from whoever actually wrote it (RHL, we think) to provide the rest. In particular: - Additional detail should be filled in in the class Doxygen for GaussianCentroidAlgorithm, in GaussianCentroid.h - The ""noPeak"" flag field description and name should be compared to what the algorithm actually does with it. It looks to me like it's a bit misnamed (and maybe shouldn't be considered an error condition at all, if we want to run this on difference images), but I'm not sure.",1 DM-1015,"convert GaussianFlux to use shape, centroid slots","We should cleanup and simplify the GaussianFlux algorithm to simply use the shape and centroid slot values instead of either computing its own or having configurable field names for where to look these up.",1 DM-1016,"Detailed documentation for SdssCentroid","We needed detailed documentation for how SdssCentroid actually works. I think it involves fitting something like a symmetrized PSF model, but I'm sufficiently unsure of the details that RHL should write this, not me. It should go in the Doxygen class docs for SdssCentroidAlgorithm in SdssCentroid.h in meas_base.",1 DM-1017,"fix testForced.py","testForced.py is currently passing even though it probably should be failing: it's trying to get centroid values from a source which has neither a valid centroid slot or a Footprint with Peaks (I suspect because transforming a footprint might remove the peaks). Prior to DM-976, that would have caused a segfault; on DM-976, I've turned it into an exception, which is then turned into a warning by the measurement framework.",2 DM-1018,"Fix incorrect eupspkg config for astrometry_net","The clang patch from 8.0.0. version was (correctly) deleted. However, the patch identity was still left in the eupspkg config's protocol. This will delete the last vestige of the formerly necessary clang patch.",2 DM-1022,"fix warnings related to libraries pulled through dependent package","This came up during migrating qserv to the new logging system, and it can be reproduced by taking log4cxx, see DM-983, essentially: {code} eups distrib install -c log4cxx 0.10.0.lsst1 -r http://lsst-web.ncsa.illinois.edu/~becla/distrib -r http://sw.lsstcorp.org/eupspkg {code} cloning log package (contrib/log.git), building it and installing in your stack, and finally taking the branch u/jbecla/DM-207 of qserv and building it. The warnings looks like: {code}/usr/bin/ld: warning: libutils.so, needed by /usr/local/home/becla/qservDev/Linux64/log/1.0.0/lib/liblog.so, not found (try using -rpath or -rpath-link) /usr/bin/ld: warning: libpex_exceptions.so, needed by /usr/local/home/becla/qservDev/Linux64/log/1.0.0/lib/liblog.so, not found (try using -rpath or -rpath-link) /usr/bin/ld: warning: libbase.so, needed by /usr/local/home/becla/qservDev/Linux64/log/1.0.0/lib/liblog.so, not found (try using -rpath or -rpath-link) {code} and they show up when I build qserv package, and are triggered by the liblog. I suspect sconsUtils deal with that sort of issues, but since we have our own scons system for qserv it is not handled. Fabrice, can you try to find a reasonable solution for that? Thanks!",1 DM-1026,"make use of MeasurementDataFlags","We started adding a system to allow algorithms to declare what kind of data they can run on, but never really put it in place. To do this, we should: - Add more flags (at least NO_CALIB). - Pass these flags in pipe_tasks and other places with tasks that use measurement tasks as subtasks (for instance, we should set NO_WCS and NO_CALIB during calibrate.initialMeasurement, and COADD in processCoadd). - Add checks for these flags in appropriate algorithms. For instance, PsfFlux should fail in construction if NO_PSF is set, and PeakLikelihoodFlux should fail if PRECONVOLVED is not set.",6 DM-1028,"qserv-version.sh produces incorrect version number","I have just installed qserv on a clean machine (this is in a new virtual machine running Ubuntu12.04) which got me version 2014_07.0 installed: {code} $ eups list qserv 2014_07.0 current b76 $ setup qserv $ eups list qserv 2014_07.0 current b76 setup $ echo $QSERV_DIR /opt/salnikov/STACK/Linux64/qserv/2014_07.0 {code} but the {{qserv-version.sh}} script still thinks that I'm running older version: {code} $ qserv-version.sh 2014_05.0 {code}",2 DM-1029,"""source"" command is not in standard shell","{{qserv-start.sh}} script fails when installed on Ubuntu12.04: {code} $ ~/qserv-run/2014_05.0/bin/qserv-start.sh /home/salnikov/qserv-run/2014_05.0/bin/qserv-start.sh: 4: /home/salnikov/qserv-run/2014_05.0/bin/qserv-start.sh: source: not found /home/salnikov/qserv-run/2014_05.0/bin/qserv-start.sh: 6: /home/salnikov/qserv-run/2014_05.0/bin/qserv-start.sh: check_qserv_run_dir: not found {code} It complains about {{source}} command. {{source}} is not standard POSIX shell command, it is an extension which exists in many shells. Apparently in older Ubuntu version {{/bin/sh}} is stricter about non-standard features. To fix the script one either has to use standard . (dot) command or change shebang to {{#!/bin/bash}}. This of course applies to all our executable scripts.",2 DM-1030,"W15 Refactoring of Qserv","Refactoring Qserv code to make it cleaner, better, and most importantly more robust and resilient to failures. JK: Refer to loading spreadsheet for PMCS assignments",84 DM-1031,"W15 Integration Testing of Qserv","Improvements to the integration tests suite for Qserv, including making it more generic (it currently uses a hardcoded database name), integrating new partitioner, and adding a data set from one of the last data releases. JK: Refer to loading spreadsheet for PMCS assignments",19 DM-1033,"W15 Design Metadata Store for production tracking (v1)","Come up with a design of Metadata Store that will track all information in the Science Data Archive (across Data Releases), for both image and database repositories JK: Refer to loading spreadsheet for PMCS assignments",45 DM-1034,"W15 Initial version of image and file archive","Build initial (alpha) version of system for tracking existing image data sets. (a) Build a web form where users enter information about existing data sets. (b) Design and build a MySQL backend (maybe through DataCat). (c) Implement crawler that automatically fetches metadata from FITS headers for data set registered by users. Note that support for additional formats (eg config files) will be added in the future, not through this epic. Note that this initial version will not be production ready, we will make it more bullet-proof and feature rich during S15 cycle. JK: Refer to loading spreadsheet for PMCS assignments",30 DM-1035,"W15 Butler (v2) for local data sets","Improvements and tweaks to the butler as needed based on the feedback from the Apps team JK: Refer to loading spreadsheet for PMCS assignments",19 DM-1036,"W15 Management of distributed Qserv databases and tables","This includes tasks such as creating and deleting distributed databases managed by Qserv, creating and deleting partitioned tables based on the information that is stored in the CSS. Design and exploratory prototype. JK: Refer to loading spreadsheet for PMCS assignments",39 DM-1037,"W15 Loading data into distributed Qserv-managed databases","Improvements to the existing data loader, to simplify data loading into Qserv. That includes integration with information in the CSS. This version will include non-parallel, single-node loader. Relevant document (describing a sketch of much more advanced version): https://dev.lsstcorp.org/trac/wiki/db/Qserv/DataLoading JK: Refer to loading spreadsheet for PMCS assignments",28 DM-1038,"S15 Implement Query Mgmt in Qserv","Initial version of system for managing queries run through qserv. This includes capturing information about queries running in Qserv. Note, we are not dealing with query cost estimate here, (it will be covered through DM-1490).",40 DM-1041,"eliminate confusing config side-effects in CalibrateTask","CalibrateTask does some unexpected things differently if you configure it certain ways, because it perceives certain processing as only being necessary to feed other steps. In particular, if you disable astrometry and photometric calibration, it only runs measurement once, because it assumes the only purpose of the post-PSF measurement is to feed those algorithms. This (as well as poor test coverage) made it easy to break CalibrateTask in the case where those options are disabled a few branches back. After conferring with Simon and Andy, we think the best solution is to remove this sort of conditional processing from CalibrateTask, which should also make it much easier to read. Instead, we'll always do both the initial and final phase of measurement, even if one of those phases is not explicitly being used within CalibrateTask itself.",1 DM-1045,"Create a permanent and accessible mapping of the BB# and the bNNN. ","Create a permanent and accessible mapping of the BB# and the bNNN. The users are interested in the BB# since is is used to point to the STDIO file form the entire stack build. The bNNN is needed because the daily life of the developer revolves around the stack tagged alternately by the bNNN tags and/or the DM Release tags. ",2 DM-1046,"W15 Central State System, support for db/table/query metadata","Improvements to the CSS system to make it more thread safe, more compact representation of keys, and adding information that will be needed to handle distributed databases, tables and likely queries (if we decided to use CSS for Query management). JK: Refer to loading spreadsheet for PMCS assignments",49 DM-1051,"W15 Image Cutout Service","First version of the images cutout service. It will rely on butler. The bulk of the work in this epic includes building a web-based front-end. JK: Refer to loading spreadsheet for PMCS assignments",19 DM-1052,"Add / improve tests and examples for Log package",NULL,1 DM-1054,"init.d/qserv-czar needs LD_LIBRARY path","With the addition of log we now need to find some shared libraries from stack. Current version of qserv-czar init.d script does not capture LD_LIBRARY_PATH, so we should add it there. ",1 DM-1055,"Remove unnecessary pieces from qserv czar config","The config file for the qserv czar has some items that are no longer relevant, and in this issue, we focus on the ones that are clearly the responsibility of our qserv css. This ticket includes: -- removing these items from the installation/configuration templates -- removing these items from sample configuration files -- removing these items from the code that reads in the configuration file and sets defaults for these items -- fixing things that seem to break as a result of this cleanup. danielw volunteers to assist on the last item, as needed. ",2 DM-1058,"fix SubSchema handling of ""."" and ""_""","SubSchema didn't get included in the rest of the switch from ""."" to ""_"" as a field name separator. As part of fixing this, we should also be able to simplify the code in the slot definers in SourceTable.",1 DM-1059,"track down difference in SdssShape implementation","The meas_base version of SdssShape produces slightly different outputs from the original version in meas_algorithms, but these should be identical. We should understand this difference rather than assume its benign just because it's small.",2 DM-1060,"S15 Data Distribution & Replica Mgmt Design","This is a continuation of DM-779, continue research related to data distribution and replication. There are still many option questions: should it be centralized or peer-to-peer? Is bittorent useful? How much should be home grown vs off-the-shelf (and which off-the-shelf). There are no direct deliverables in FY2015, but it is a complex topic, so we should not wait until the last minute. This epic covers the R&D / design.",100 DM-1061,"W15 Qserv Stability / bug fixes","Tweaks, improvements, fixes to bugs discovered over the course of W15, to improve Qserv stability. JK: Refer to loading spreadsheet for PMCS assignments ",83 DM-1067,"move algorithm implementations out of separate subdirectory","We should move the code in the algorithms subdirectory (and namespace) into the .cc files that correspond to individual algorithms. They should generally go into anonymous namespaces there. After doing so, we should do one more test to compare the meas_base and meas_algorithms implementations.",1 DM-1068,"audit and clean up algorithm flag and config usage","Check that meas_base plugins and algorithms have appropriate config options and flags (mainly, check that there are no unused config options or flags due to copy/paste relics).",1 DM-1070,"switch default table version to 1","Now that all tasks that use catalogs explicitly set the table version, it should be relatively straightforward to set the default version to 1 in afw. Code that cannot handle version > 0 tables should continue to explicitly set version=0.",2 DM-1071,"Switch default measurement tasks to meas_base","We should set the default measurement task in ProcessImageTask to SingleFrameMeasurementTask, and note that SourceMeasurementTask and the old forced photometry drivers are deprecated.",2 DM-1072,"create forced wrappers for algorithms","We have multiple algorithms in meas_base which could be used in forced mode but have no forced plugin. We should go through the algorithms we have implemented and create forced plugin wrappers for these.",1 DM-1073,"remove old forced photometry tasks","After meas_base has been fully integrated, remove the old forced photometry tasks from pipe_tasks",1 DM-1074,"Measurement - Calibration and Ingest","Create command-line tasks to transform raw measurement quantities (e.g. fluxes, positions in pixels) to global units (e.g magnitudes and positions in celestial coordinates). Also requires interfacing with the measurement plugin system, as only plugins can be authoritative on how to transform their outputs. JK: In PMCS this would be Bosch J 50% Gee P 50%",20 DM-1076,"convert afw::table unit tests to version 1","Most afw::table unit tests explicitly set version 0. We should change these to test the new behaviors, not the deprecated ones.",2 DM-1077,"Audit TCT recommendations to ensure that all standards updates were installed into Standards documents.","Audit TCT recommendations to ensure that all standards updates were installed into Standards documents. It was found that the meeting recorded in: [https://dev.lsstcorp.org/trac/wiki/Winter2012/CodingStandardsChanges] failed to include two recommendations: * recommended: 3-30: I find the Error suffix to be usually more appropriate than Exception. ** current: 3-30. Exception classes SHOULD be suffixed with Exception. * recommended but not specifically included: Namespaces in source files: we should use namespace blocks in source files, and prefer unqualified (or less-qualified) names within those blocks over global-namespace aliases. ** Rule 3-6 is an amalgam of namespace rules which doesn't quite have the particulars desired. FYI: The actual vote was to: ""Allow namespace blocks in source code (cc) files."" To simplify the future audit, all other recommendations in that specific meeting were verified as installed into the standards.",2 DM-1078,"add batch flag to newinstall.sh","In some cases (installing in some script environment) it is nice to be able to run without any interaction. This will add a flag to tell the script to install without asking about anaconda and git. It will assume the answer is 'yes' to any question it would have asked.",1 DM-1079,"Organize the content of the DM Developer Guide","The DM Developer Guide is presently a place-holder for information relevant to developers. It is time to re-organize the content, and to provide authoritative guidance to developers. This will involve contributions from various members of the DM Team, and likely some selected harvesting of the Trac/Wiki. ",10 DM-1083,"Fix overload problems in SourceCatalog.append and .extend","This example fails with an exception: {code:py} import lsst.afw.table as afwTable schema = afwTable.SourceTable.makeMinimalSchema() st = afwTable.SourceTable.make(schema) cat = afwTable.SourceCatalog(st) tmp = afwTable.SourceCatalog(cat.getTable()) cat.extend(tmp) {code} Expected behavior is that the last line is equivalent to {{cat.extend(tmp, deep=False)}}.",1 DM-1085,"Update the DMS and Astro Glossaries","The DMS and Astro Glossaries in Confluence define a set of technical terms used in their respective domains. Some of the definitions are placeholders, and other terms used in the SWUG, DM Space, and DM Developer Guide have yet to be defined in one glossary or the other. It is time to update these docs.",2 DM-1086,"Determine problem with Mac OS X processCCD output when compared to 'identical' dataset generated on RHEL6","The benchmark file for the summer2012 demo is generated on RHEL6. When using the dataset to compare stack output generated on Mac OSX 10.8.5 (and 10.9) there are significant deviations. Find out where the problem arises...in the stack during processing or in the comparison.",4 DM-1087,"Research how to kill query in mysql","In the port to the new Xrootd Ssi API, worker-side squashing was lost in the shuffle. The plumbing is different, and re-implementing squash functionality is not entirely straightforward, especially because the new API is still missing documentation and examples for implementing cancellation. The consequences of not implementing this are minor--some extra work may be done by the worker, but not a whole lot, because user-level cancellation has not been implemented. First step is to understand how mysql code is handling query killing.",4 DM-1088,"Investigate HTCondor config settings to control speed of ClassAd propagation","With default settings we do not have good visibility as to whether an updated ClassAd on a compute node (e.g., CacheDataList now has ccd ""S00"") will be in effect on the submit node in time for a Job to be matched to an optimal HTCondor node/slot. There are several components (negotiator, schedd, startd) and their associated activities that could impact the time that it takes for a new ClassAd on a worker node to 'propagate' back to the submit side. We investigate these configuration settings to try to determine what thresholds for configuration settings are required to meet a given time cadence of job submissions.",2 DM-1089,"Enhance SWUG chapter on measurement","A basic description of measurement in the LSST Stack was created for the SWUG (see DM-692). However, this chapter lacks specifics about how the measurement algorithms work and does not yet mention some of the available algoritms. This task is intended to provide the next level of detail, with references to source-level descriptions that exist or are in progress. ",6 DM-1099,"afw::table - finish interface transition","JK: In PMCS Bosch J and Gee P",36 DM-1100,"Measurement - Convert Old Algorithms","JK: In PMCS Bosch J and Gee P",31 DM-1101,"Measurement - Finish Framework Overhaul","JK: In PMCS Bosch J and Gee P",51 DM-1103,"Design unit-test data package","Gather requirements for unit test datasets and determine how to organize and generate the data.",4 DM-1104,"Generate test data","Write scripts to generate the test data (as needed) and run them.",10 DM-1105,"Custom mapper for unit test data package","We need a custom mapper for the test data package, so we can run unit tests against it without depending on any particular obs_* package.",6 DM-1106,"Convert existing tests to use new package","After the new unit test data package is available, we should convert all tests that currently use afwdata or obs_test to use the new package instead, and then remove obs_test and afwdata. Should have subtasks and a re-estimate of effort after auditing how much there is to be done.",30 DM-1107,"afw - Footprint Improvements","This epic contains several improvements to the Footprint class, including API refactoring, performance-related reimplementations, and some new features. JK: In PMCS this would be Bosch J and Swinbank J",30 DM-1108,"Galaxy Fitting - Shear Precision Experiments","NB some of the below stories will actually be Lauren, not Perry. Breakdown: pgee 58%; jbosch 21%; krughoff 6%; lauren 15%",100 DM-1109,"Minimal MultiFit system prototype","Experiment with a framework for running ""multifit""-style fitting algorithms (i.e. fit a model convolved with the PSF to multiple exposures simultaneously). Earlier prototypes (meas_multifit) have demonstrated executable multifit code on a single system: repeating that work is not the priority here. Rather, we will produce a concept for the framework for running multifit at scale using large datasets on a cluster. It is not a requirement that it be possible to run an at-scale multifit job and produce useful results by the end of this epic; however, there should be proof-of-concept code which demonstrates the structures within which future multifit work will be carried out. This will be supplied to both Science Pipelines and Process Middleware developers.",100 DM-1110,"Galaxy Fitting - Sampling Algorithm Plugin",NULL,60 DM-1111,"CModel robustness","The CModel code is currently a mixture of work ported from HSC and some new development on LSST. It needs to be validated on real data (HSC, SDSS, CFHT). All known failure modes should be eliminated. Where possible, improve its performance.",65 DM-1112,"Create utilities to allow camera testing team to use CameraGeom","The camera team can use the CameraGeom classes to reduce lab data for testing purposes. Since the camera is relatively flexible, a good way of doing this is to create a camera at runtime from the header keys in the files to be reduced. JK: In PMCS this would be Krughoff S",30 DM-1113,"Make the API for ISR explicit","The run method of the IsrTask currently takes a dataRef which has getters for calibration products. This makes the task hard to re-use because one needs a butler and because the interface is opaque. This task will make the IsrTask API more transparent. JK: In PMCS this would be Krughoff S",20 DM-1114,"Re-write astrometry task to remove dependency on astrometry.net","The astrometry task currently depends as astrometry.net as the only possible algorithmic back end. This also has created a dependency on the astrometry.net file format. This task will remove the dependency on astrometry.net file format and will implement the current HSC astrometry solver in the astrometry task. This will focus on solving single chips, not the full focal plane. Note that DM-167 (which was closed with reference to this ticket) requests that we split photometric and astrometric catalogues, and this is still needed. JK: In PMCS this would be Krughoff S and Owen R",80 DM-1115,"Command-line Tasks - Unit tests for ProcessCcd","We can get a long way toward improved test coverage in pipe_tasks by having Unit tests for ProcessCcd. JK: In PMCS this would be Owen R",20 DM-1116,"Add numpy typemaps to afw interfaces so numpy types pass correctly to C++","Currently SWIGed interfaces do not handle numpy types correctly. This can be fixed with the appropriate typemap. JK: In PMCS this would be Owen R",11 DM-1117,"Extend Exposure classes to contain background models","Exposures carry many things: PSFs, FITS metadata, Detectors, etc. Another object algorithms may want to access with an exposure is the set of background models subtracted from the exposure to that point. This will involve porting afw.math.BackrgoundList to C++ and extending Exposure classes to contain the new BackgroundList object. JK: In PMCS this would be Krughoff S",20 DM-1118,"Improve documentation of code from DM-70","DM-70 was merged before its code documentation was completely ready, in an effort to not leave such a large amount of code un-merged. This ticket exists as the second half of DM-70 to clean up its documentation.",6 DM-1120,"Improve fault tolerance as demonstrated in nightly computing simulator","Adjust simulator parameters, mechanisms, and implementation to minimize time to recover and unprocessed/unarchived data resulting from a machine or network failure. For W15: Pietrowicz, S - 100% Start Oct 2014, finish Nov 2014 For S15 rollover: Pietrowicz, S - 12% Start March 2015, finish April 2015",50 DM-1121,"Simplify and refactor the Event Services API","Remove old fixed metadata; refactor using more generic metadata interface. For W15: Pietrowicz, S - 100% Start Dec 2014, finish Dec 2014 For S15 rollover: Pietrowicz, S - 11% Start March 2015, finish April 2015 ",18 DM-1122,"Write a log4cxx handler that emits events for log messages","Write a log4cxx handler that emits events for log messages. For W15: Pietrowicz, S - 100% Start Jan 2015, end mid Jan 2015 For S15 rollover: Pietrowicz, S - 11% Start March 2015, finish April 2015",18 DM-1123,"requirements and specifications for initial version of inter-Task communication","gather requirements and specification for Implementation of a gather/scatter mechanism for a set of Tasks executing in parallel using the Event Services, providing for eventual extension to support MPI or other communication mechanisms. Start mid Jan 2015, finish Feb 2015. JK: In PMCS this would be Pietrowicz S",50 DM-1124,"Chebyshev approximation object for aperture corrections","Using the interface defined in DM-740, implement a Chebyshev-based implementation to be used to interpolate aperture corrections across CCD-level interfaces. A prototype is available on the HSC fork, which can be copied directly, modulo any interface changes on DM-740. For more information, see the HSC Jira issue (which also includes the work associated with DM-740): https://hsc-jira.astro.princeton.edu/jira/browse/HSC-796 and the HSC git commits https://github.com/HyperSuprime-Cam/afw/compare/releases/S14A_0...tickets/DM-796",3 DM-1125,"avoid usage of measurement framework in star selectors","At least one of the star selectors uses the old measurement framework system to measure the moments of a cloud of points. With the new versions of all the measurement plugins, it should be much easier (and cleaner) to just call the SdssShape algorithm directly, instead of dealing with the complexity of applying the measurement framework to something that isn't really an image.",3 DM-1126,"design new Footprint API","This issue is for *planning* (not implementing) some changes to Footprint's interface, including the following: - make Footprint immutable - create a separate SpanRegion class that holds Spans and provides geometric operators does not hold Peaks or a ""region"" bbox (Footprint would then hold one of these). - many operations currently implemented as free functions should be moved to methods - we should switch the container from vector to simply vector, as Span is nonpolymorphic and at last as cheap to copy as a shared_ptr. The output of this issue will be a set of header files that define the new interface, signed off by an SAT design review. Other issues will be responsible for implementing the new interface and fixing code broken by the change.",8 DM-1127,"implement new Footprint API","This issue is the implementation for DM-1126, including fixing code broken by the API changes. This issue should be given subtasks for discrete pieces of work as part DM-1126, as it has a lot of story points. Because it's likely all of this needs to be merged at the same time, it should probably be done on a single branch, unless some of the earlier work can be done in a backwards-compatible way..",20 DM-1128,"Span-based grow operations for Footprint","The current grow operation for Footprints is very inefficient for isotropic grows. A better algorithm can be found in the attached paper.",10 DM-1129,"Span-based topological set operations for Footprint","Implement span-based overlap tests and spatial union, intersection, and difference operators for Footprints. Should be split into subtasks; first task would be to come up with the interface, and then each operation to be implemented should have one subtask each. If complete after DM-1126 and DM-1127, these operations should be implemented in the SpanRegion class, and Footprint should delegate to that.",20 DM-1130,"refactor C++ Algorithm classes","Using one of the prototypes of DM-828, refactor the C++ Algorithm code in meas_base and any other packages.",10 DM-1131,"create PSF simulations for shapelet approximation test","To determine the number and order of shapelet expansions needed to approximate the LSST PSF, we need simulations of LSST PSFs, presumably generated using PhoSim. I think we want something like 50-100 extremely high-SNR stars, drawn from realistic distributions of anything that would affect the PSF, including position on the focal plane and center position within a pixel. I only care about getting these out as postage stamps, so I'll leave the question of how best to run PhoSim to get these up to someone else. While this is part of a mostly-Princeton Epic, I'm assigning this issue to Simon as he'd be able to do it much faster than I could. I'll also let him review and update the story points estimate.",6 DM-1132,"create galaxy simulations for shapelet approximation and truncation tests","Using the PSF simulations generated in DM-1131 and the GalSim package, generate ~20k galaxies for each of 3 shear values for each input PSF. Will need to be divided into subtasks - figure out what the outputs should look like, write the code to generate the simulations, estimate the time need to run the simulations, run the simulations at scale, etc. ",30 DM-1133,"implement shear measurement driver for simulations","We'll need driver code to run the galaxy fitting algorithms on the simulations from DM-1132. Efficiently saving all the samples could be challenging.",10 DM-1134,"implement SDSS PSF residual trick","SDSS galaxy fitting approximated the convolution of a galaxy model with the PSF as the convolution of the galaxy model with a simplified approximation to the PSF added to the difference between the PSF approximation and the true PSF. We should do the same, as it'll be no worse than ignoring the difference between the PSF approximation and the true PSF, and it may greatly reduce the complexity needed in the approximation.",8 DM-1135,"test how large pixel region used in galaxy fitting needs to be","Using simulations built on DM-1132 and driver code from DM-1133, test different pixel region sizes and shapes, and determine at what point shear bias due to finite fit region drops below a TBD threshold.",20 DM-1136,"test number/order of shapelet expansions needed to approximate PSF","Run driver from DM-1133 on simulations from DM-1132 with different configuration for shapelet PSF approximation, increasing complexity until change in shear estimate drops below a TBD threshold.",14 DM-1137,"Evaluate python/c++ documentation generation and publication tools ","This epic related to documentation that is provided as part of normal development activities. The desire is to keep this documentation in and near the codebase as this is best practice for it being maintainable. At the other end, we wish to publish this documentation in a coherent and searchable way for users. A number of tools exist in this area and this item requires a preliminary evaluation to be made. This is part of curating our documentation infrastructure. [FE 75% DOC 100% starting August 20th] ",20 DM-1138,"Demonstrate & iterate with team on documentation toolchain ","Following from DM-1137, this epic relates to demonstrating various options for documentation tools workflows to the team, gathering input as to the preferred solution, adopting a workflow, and defining any specific implementation choices. This is part of curating our documentation infrastructure. ",5 DM-1139," Stack documentation infrastructure and migration","[Epic retitled and bumped in points to reflect wider scope and change in resource allocation] This epic containes work on migrating the documentation infrastructure into sphinx (from doxygen, Confluence) subject to RFC, continuous deployment of documentation, read-the-docs or similar presentation, standardisation via teplates of common information, a proposal for CI-ing examples and tutorials, release and install note documentation, and visual design and javascript development for UI/UX An additional request to migrate Word-based design documentation has been accepted. LaTeX support is being investigated. [JS 100%] ",63 DM-1140,"Investigate automatic MacOS X build/deploy","This item is to set up the same continuous integration process we have on Linux on a MacOSX test server. JK: In PMCS this would be Economou F and New Hire LS3",4 DM-1141,"Evaluate merits of alternative CI and RFC","Evaluate whether we wish to continue buildbot development or use a different (or additional) continuous intergration system. This is part of a continuous occasional process of evaluating whether our current toolchain choices are still meeting our needs. [FE at 75%, JH at 75%]",30 DM-1142,"Regularise Nightly and Weekly builds ","The intent here is to create two seperate automated deployment environments, one based on a nightly (or ad-hoc) build, one one aslower cadence (eg weekly). This will allow us to do intergration/QA runs on a bleeding or trailing edge as required. JK: In PMCS this would be Economou F and New Hire LS3",6 DM-1143,"Investigate candidates for Verification and Integration Data Sets","The task here is to develop a data set that can be used both for continuous integration (build tests) and automatic QA (integration tests). We want to maximise the richness of the data set in terms of its usefulness, but minimise it in terms of its size. DN to co-ordinate contributions. [DN 95% FE 5%]",40 DM-1145,"Remove code made obsolete",NULL,4 DM-1147,"Create a top-level qserv_distrib package","qserv_distrib will be a meta-package embedding qserv, qserv_testdata and partition.",2 DM-1148,"SUI: Research system framework for SUI development","Current IPAC development utilizes GWT, is it the right system for LSST SUI in 2022? I think this will be an on-going activity for the first two years. 10% of Goldina, Zhang, Ciardi, Surace 20% of Roby, Rector, Ly, Wu 40% Groom",100 DM-1149,"W15 Metadata Store for production tracking (prototype)","Build a first prototype of the Metadata Store. JK: Refer to loading spreadsheet for PMCS assignments",45 DM-1150,"Fast image search","A lot of users will search for images using some (often advanced) spatial criteria. Need to implement something similar to what we have in UDFs/SciSQL to efficiently support this class of search. Check GIS support in MariaDB",10 DM-1151,"Fix example of IsrTask to be callable with data on disk","Currently the example of the IsrTask takes a fake dataref. This is hard to use with real data. In DM-1113 we will update IsrTask to not take a dataRef. This will make it easy to update the example script to work with real data. This ticket will also include removing from the unit tests any fake dataRefs that have become unnecessary as a result of DM-1299. ",2 DM-1152,"Css C++ client needs to auto-reconnect","The zookeeper client in C++ that the czar uses doesn't auto-reconnect. This is a capability provided in the kazoo library that qserv's python layer provides, but isn't provided in the c++ client. The zookeeper client disconnects pretty easily: if you step through your code in gdb, the zk client will probably disconnect because its threads expect to keep running. zk sessions may expire too. Our layer should reconnect unless there is really no way to recover without assistance from the calling code (e.g. configuration is wrong, etc.). This ticket includes only basic reconnection attempting, throwing an exception only when some ""reconnection-is-impossible"" condition is met.",2 DM-1153,"Minor problems in lsstsw, related to Qserv offline install procedure","- on lsstsw master branch tip, ./stack/Linux64/lsst/9.2/bin/newinstall.sh doesn't seems to be the last version (it still install eups-1.3.0 instead od eups-1.5.0) - on Fedora19, flock from util-linux 2.23.1 doesn't support next options : {code:bash} $ flock -w 0 200 flock: timeout cannot be zero {code} but {code:bash} flock -w 1 200 {code} works, - newinstall.sh : would it be possible to enable automatic answers to git and anaconda install questions. For example, in order to easily enable automatic install on 300 nodes clusters ? (cancelled : covered by DM-1078) - loadLSST.sh appends automatically http://sw.lsstcorp.org/eupspkg to EUPS_PKGROOT, and if first url in EUPS_PKGROOT isn't available eups fails without trying next ones => this isn't compliant with offline mode and introduce a work-around in Qserv offline mode install scripts. Would it be possible to define a lightly different behaviour for loadLSST.sh ? - In newinstall.sh, l. 167, if Python version isn't correct, then exit *with error code*. ",3 DM-1154,"SUI web user interface prototype ","SUI team want to use this time to study the DM system and all the other design documents to come up with preliminary design of the SUI infrastructure with some prototyping along the way to help the proof of concept. 40% Wu 30% Rector, Roby, Ly 10% Zhang, Goldina 80% Ciardi 90% Surace 60% Groom",100 DM-1155,"SUI Interface through Qserv with database and prototype ","Working closely with database group to define the interface needs between SUI and Qserv. Prototyping the functions will help us understand the interface better. 50% Rector 10% Ly 10% Wu",32 DM-1156,"SUI: Query and display LSST image ","Define the APIs for image query with database group. Depend on the implementation of APIs Exercise the image cutout service DM-1977 been developed in SLAC. ",6 DM-1157,"SUI catalog query interface prototype","Define the APIs for image query with database group Depend on the implementation of APIs 30% Goldina 20% Ly 10% Wu",34 DM-1158,"SUI image visualization prototype (without searching LSST images)","Using the current software components developed in IPAC to put together a prototype of visualization capabilities. The purpose is to get feedback from DM people and potential users of the tool. 20% Roby 30% Zhang",34 DM-1159,"SUI 2D plot for catalog prototype","Using the current software components developed in IPAC to put together a prototype of visualization capabilities. The purpose is to get feedback from DM people and potential users of the tool. 30% Goldina 10% Ly",38 DM-1160,"SUI catalog and image interactive visualization with LSST data","Using the current software components developed in IPAC to put together a prototype of visualization capabilities. The purpose is to exercise the data access APIs developed by SLAC and get feedback from DM people and potential users of the tool. 20% Goldina, Zhang 10% Roby, Ly, Wu, Ciardi ",20 DM-1161,"Cleanup SdssShape","We should do a comprehensive cleanup of the SdssShapeAlgorithm class. This includes removing the SdssShapeImpl interface (never supposed to have been public, but it became public) from other code that uses it, and integrating this code directly into the algorithm class. We should also ensure that the source from which the algorithm is derived is clearly cited -- that's Bernstein and Jarvis (2002, http://adsabs.harvard.edu/abs/2002AJ....123..583B); see also DM-2304.",8 DM-1162,"Port meas_algorithms unit tests for plugins","While test coverage isn't complete, there are many unit tests for measurement plugin algorithms in meas_algorithms that have not yet been ported to meas_base. We should make sure any of these tests that aren't already covered in meas_base are moved over before we remove the meas_algorithms versions of things.",6 DM-1164,"remove dependency between sconsUtils and eups","sconsUtils currently depends on eups, (and as far as I understand it, it should not...)",4 DM-1167,"experiment with building MyISAM as a shared library","Per Monty (phone call Aug 28, 2014), this should be easy, code to look at is in storage/myisam/mi_test*",2 DM-1168,"zookeeper port numbers should be configurable","The test programs in core/modules/css/ has hardcoded zookeeper port numbers. That needs to be fixed, it needs to be configurable.",1 DM-1169,"User-friendly install/configure/test scripts","- stdout, stderr output should be colorized, whereas redirecting it into a file shouldn't.",3 DM-1170,"FY19 Add Support for ForcedSource Table in Qserv","ForcedSources will needs special handling * ra/decl columns are not part of ForcedSource table, we will need them for partitioning * we might want to store them in subchunks? ",53 DM-1176,"Make default image origin PARENT in all cases","After DM-840 is finished, and code updated for it (DM-846), make the new default image origin PARENT, remove the UNDEFINED image origin enum, and update at least some of the code that explicitly specifies PARENT (there is little harm in leaving this explicit, other than it does not set an optimal example).",4 DM-1177,"Implement sharable just-in-time subchunk management on the worker","The current subchunk management on the worker is very simple. When a query fragment is selected for execution, it builds exactly the subchunks it needs, performs the queries, and destroys the subchunks. This presents a problem when we have concurrent queries that need the same subchunks. Thus an earlier query will destroy its subchunks (some/all of which are needed by another concurrent query) and cause the later queries to fail. The suggested approach is as follows: * Each query, instead of building subchunks on its own, delegates responsibility for subchunk creation/deletion to another class (e.g. SubChunkManager, [but try to find a better name]). * Each query requests a subchunks and releases subchunks using a lock()/unlock() mechanism (in the SubChunkManager API), or wrapped in an interface like boost::lock_guard wraps boost::mutex where acquisition and deletion can be handled by scoping. (It might even be possible to use boost::lock_guard mechanism.) * Now SubChunkManager blocks calls to lock() until it has created the necessary subchunks and modified its records to note that these subchunks are needed by one additional query). A subsequent unlock() call from the client indicates that SCM can decrement its counter and destroy the appropriate subchunks. I (danielw) think the data structure to manage this is straightforward and the code for creation/deletion of subchunks already exists, but it might not be obvious how to route the plumbing so that queries can request/release appropriately, to an instance (probably only one per worker instance) owned by an object somewhere else. Please feel free to use a better or more maintainable approach. ",15 DM-1185,"Channel was inactive for too long error","There as an issue with the receiveEvent call that can cause an exception ""Channel was inactive for too long"". The fix for this (according to the ActiveMQ users list and archives) is to either completely disable the inactivity monitor or to increase the inactivity limit to something extremely high. The fix is adding: ""wireFormat.maxInactivityDuration=0"" to the URL in establishing the connection. There might also be a way of doing this directly in the ActiveMQ broker, but so far I haven't seen anything that would let me specify that.",2 DM-1187,"Learn the OCS middleware","OCS will deliver the OCS Middleware software in November 2014. There will be a workshop held at SLAC the week of November 10th. I spoke with K-T and we should be able to attend this remotely, if necessary. We need to spend some time getting familiar with this software and how it will integrate with the Base DMCS for AP.",6 DM-1188,"rewrite low-level shapelet evaluation code","While trying to track down some bugs on DM-641, I've grown frustrated with the difficulty of testing the deeply-buried (i.e. interfaces I want to test are private) shapelet evaluation code there. That sort of code really belongs in the shapelet package (not meas_multifit) anyway, where I have a lot of similar code, so on this issue I'm going to move it there and refactor the existing code so it all fits together better.",2 DM-1192,"Write a transition plan to move gitolite and Stash repositories to GitHub","As recommended by the SAT meeting on 2014-09-16, we need this document to promote the use of GitHub by other subsystems within the project and to understand the impacts on DM. The plan should include, but is not limited to: * Whether and how the repositories should be reorganized. * How existing commit attributions will be translated. * Moving comments in Stash to GitHub",20 DM-1195,"There is a bug in the prescan bbox for megacam.","The bounding box of the prescan region in the megacam camera should have zero y extent (I think). Instead it goes from y=-1 to y=2. This is either a bug in the generation of the ampInfoTables or in the way the bounding boxes are interpreted.",1 DM-1196,"exampleUtils in ip_isr is wrong about read corner","https://dev.lsstcorp.org/cgit/LSST/DMS/ip_isr.git/tree/examples/exampleUtils.py#n95 Says that the read corner is in assembled coordinates. This is not true, it is in the coordinates of the raw amp. That is, if the raw amp is in electronic coordinates (like the lsstSim images) it is always LL, but if it is pre-assembled, it may be some other corner. This should probably use the methods in cameraGeom.utils to do the image generation.",1 DM-1197,"Support some mixed-type operations for Point and Extent","The current lack of automatic conversions in python is pretty irritating, and I think it's a big enough issue for people writing scripts that we should fix it. In particular, allow {code} Point2D + Extent2I Point2D - Extent2I Point2D - Point2I Extend2D + Extent2I Extend2D - Extent2I {code} (and the respective operations in the opposite order where well defined) It would also be good to allow the all functions expecting PointD to accept PointI, but I'm not sure if swig makes this possible. It's probably not worth providing C++ overloads for all of these functions (and to be consistent we should probably do all or none). I realize that you invented these types to avoid bare 2-tuples, but I'm not convinced that we shouldn't also provide overloads to transparently convert tuples to afwGeom objects.",2 DM-1199,"Revisit log integration with Xrootd","The integration of logging with xrootd needs to be revisited, for both the czar and the worker, after a discussion with Xrootd devs about API changes. We want to accomplish 2 things: * Logging from xrootd client should go through the qserv-czar's logger, so it can be saved in the same file and enabled/disabled as a component. * Logging from the xrootd server may also find a benefit in using our logger as a backend",20 DM-1201,"add ColumnView support to FunctorKeys","FunctorKeys currently only work with individual records, but at least some could work with columns as well. Need to add an interface for this and decide how to handle cases where it the FunctorKey doesn't support it.",2 DM-1210,"Invert buffering for czar in row-based result handling","The XrdSsi API performs some impedance-matching in buffering transfers. The current code (introduced in DM-199) doesn't leverage this because its flow model is based on the older mysqldump-based results passing. With dump files, we don't know the size of the fragments expected, so we are just passing buffers of bytes and building up a text blob to ingest. With the row-based protocol, we have sizes specified, hence the receiver can specify exactly what sizes of bytes are needed. Implementation of this ticket should simplify and reduce the code overall. ",10 DM-1211,"anaconda is too outdated to work with pip","The version of anaconda distributed with the stack is too outdated to be used with pip (and probably other things). The issue is an unsafe version of ssh. A workaround is to issue this command while anaconda is setup: {code} conda update conda {code} Warning: it is unwise to try to update anaconda itself (with ""conda update anaconda"") because that will revert some of the changes and may result in an unusable anaconda. I think what is required is an obvious change to ups/eupspkg.cfg.sh The current version of anaconda is 2.0.1 based on http://repo.continuum.io/archive/ Note: there is no component for anaconda. I will submit another ticket.",2 DM-1213,"cleanup order/grouping of header files","We want: * header for the class * then system * then third party * then lsst * then qserv We currently don't have the ""lsst"" group (with a few exceptions), and we call the last one ""local"" in most places.",1 DM-1215,"makeMaskedImage leaks memory","Calling afw.image.makeMaskedImage leaks memory when called from Python, because it returns a raw pointer without telling Swig %newobject. To fix it, it'd be better to just have it return by value instead of by pointer, though this might involve fixing some downstream C\+\+ code (Python code should not be affected).",1 DM-1216,"compute linear parameter derivatives more intelligently in optimizer","The numerical derivatives computed by the optimizer currently don't distinguish between the linear parameters (for which derivatives are trivial) and nonlinear parameters (for which they're hard), because we don't pass the information that distinguishes them to the object that computes the derivatives. If we move the computation of derivatives from the Optimizer class to the Objective class, we should be able to compute the derivatives much more efficiently. While this doesn't matter much when fitting single component galaxy models (because there's only one linear parameter in that case), it should matter quite a bit when fitting high-order shapelets to PSF models.",4 DM-1217,"Refactor meas_base Python wrappers and plugin registration","meas_base currently has a single Swig library (like most packages), defined within a single .i file (like some packages). It also registers all of its plugins in a single python module, plugins.py. Instead, it should: - Have two Swig libraries: one for the interfaces and helper classes, and one for plugin algorithms. Most downstream packages will only want to %import (and hence #include) the interface, and having them build against everything slows the build down unnecessarily. The package __init__.py should import all symbols from both libraries, so the change would be transparent to the user. - Have separate .i files for each algorithm or small group of algorithms. Each of these could %import the interface library file and the pure-Python registry code, and then register the plugins wrapped there within a %pythoncode block. That'd make the implementation of the algorithms a bit less scattered throughout the package, making them easier to maintain and better examples for new plugins.",3 DM-1218,"Support multiple-aperture fluxes in slots","We should be able to use multiple-aperture flux results in slots. While this is technically possible already by setting specific aliases, it doesn't work through the usual mechanisms for setting up slots (the define methods in SourceTable and the SourceSlotConfig in meas_base). After addressing this, we should remove the old SincFlux and NaiveFlux algorithms, as the new CircularApertureFlux algorithm will be able to do everything they can do.",2 DM-1226,"Refine Butler prototype","A prototype of the new Butler is part of the S14 delivery. This needs to be refined into a production package and the rest of the code needs to be ported to use it.",15 DM-1227,"S15 Multi-node Multi-query Integration Testing Harness","Build end-to-end integration test harness for Qserv that will run queries on multi-node system.",56 DM-1228,"LSE-68: Bring pull interface to CCB approval","Bring a long-pending set of changes, primarily the adoption of the ""pull interface"" for all DM-Camera image requests, to CCB approval",6 DM-1229,"LSE-68: Major phase-3 details (W15)","Deepen ISD to cover: supply of crosstalk constants, deletion policy details, content of the new-data notification, availability of a pass-through tag in data, and other Phase 3 matters. Edit ISD to ensure that it covers WFS and guider requirements as well. Deliverables: * Marked-up LSE-68 with combination of DM proposals for Phase 3 details and, where that is not a realistic approach, specific questions for Camera DAQ team. * Bring a proposal to the CCB by the end of the cycle.",16 DM-1230,"LSE-69: Bring Summer 2014 work to CCB approval","Bring Summer 2014 work on LSE-69 to CCB approval, ideally by 20 October in time to be part of the CD-2 document package.",6 DM-1232,"LSE-72: Bring Summer 2014 work to CCB approval","Remaining work is to proofread the SysML-ization by Brian Selvy of the LSE-72 draft, do any required cleanup in conjunction with the OCS team, and advocate for LCR-202 (already exists) at the CCB.",3 DM-1233,"Refine requirements and use cases for Level 3 facilities","Refine the requirements and use cases for the three branches of Level 3 capabilities exposed to users: * Level 3 programming toolkit (user reconfiguration / extension of DM pipelines and stack) * Level 3 compute cycle delivery (user access to 10% of compute base) * Level 3 data product storage Deliverables: * Refinement, if necessary, to Level 3 requirements in DMSR * Flowed-down requirements as a separate document. Sufficient detail to allow a breakdown of the deliverables in the three areas of Level 3 by annual release cycle through construction period.",20 DM-1234,"LSE-72: Remaining Phase 2 details","Sort out remaining Phase 2 details of LSE-72, especially: * details of the configuration mechanism (yet to appear even in LSE-70), the publication of available configuration keys, the efficient publication of configuration contents, OCS-driven configuration ""knobbing"" * specific list of events to be published by DM * more specific EFD-query language * EFD deployment model (understanding of distributed design envisioned by OCS and its implications for the ICD)",10 DM-1235,"LSE-72: Phase 3 details","Advance Phase 3 details as needed to eliminate obstacles to OCS and DM development during Summer 15.",10 DM-1236,"LSE-75: Bring Summer 2014 work to CCB","Bring a small set of technical changes (e.g., pointing notice moved to OCS ICD) and the addition of PSF reporting to T&S into the SysML version of LSE-75 and submit a change request to get this approved.",6 DM-1237,"LSE-75: Refine WCS and PSF requirements","Clarify the data format and precision requirements of the TCS (or other Telescope and Site components) on the reporting of WCS and PSF information by DM on a per-image basis. Depends on the ability of the T&S group to engage with this subject during the Winter 2015 period. Can be deferred to Summer 2015 without major impacts. Current PMCS deadline for Phase 3 readiness of LSE-75 is 29-Sep-2015.",8 DM-1238,"LSE-130: Bring Summer 2014 work to CCB approval","Bring the current list-oriented version of LSE-130 into the camera-test-plan-oriented format requested by the Camera group, moving the old list(s) into LDM-272 for reference. Negotiate what further work is required to make the document acceptable to the Camera group for CCB approval in time for CD-2. Nominally this requires CCB approval by 20-Oct-2014 in order to have an approved document properly released to the CD-2 committee.",16 DM-1239,"LSE-130: Make requests more quantitative","Proceed, as enabled by input from the DM Apps group and the Camera, to make the individual data requests in LSE-130 more quantitative.",12 DM-1240,"LSE-140: Bring Summer 2014 work to CCB approval","Convert the existing LSE-140 draft to SysML, produce a docgen, and review with Jacques Sebag. Bring to CCB meeting on 8 October under existing LCR-201.",6 DM-1241,"Complete data entry of LSE-140 revised draft into EA",NULL,2 DM-1242,"Risk Register refresh 1/2015","Periodic review of DM risk register contents. Covers preparation for a review expected at the end of January 2015, the only one during Winter 2015.",3 DM-1244,"TOWG - Contributions to the Operations Concept Document","Covers contributions to the writing and editing of the Operations Concept Document during the Winter 2015 cycle. Deliverables: * Timescales diagram and explanatory text * Framework for Nominal Operations chapter * Contributions to KTL's writing of the Data Processing Operations chapter * General proofreading",10 DM-1245,"Install scisql plugin (shared library) outside of eups stack.","sciSQL plugin is currently deployed in eups stack (i.e. $MYSQL_DIR/lib/plugin) during configuration step. Nevertheless eups stack should be immutable during configuration step. MySQL plugin-dir option may allow to deploy sciSQL plugin outside of eups stack (for example in QSERV_RUN_DIR).",3 DM-1246,"add per-exposure callbacks for measurement plugins","We should give measurement plugins an opportunity to do some work whenever we start processing a new Exposure. This give them a good time to throw exceptions when there's something obviously wrong with the exposure (e.g. it's missing a Psf). It also gives them an opportunity to do work that could be used to speed up per-source processing. One specific case I have in mind is in shapelet PSF approximation for galaxy fitting, where it could be very valuable to use an initial fit to an average PSF to initialize (and speed up) the first to per-source PSFs. One question here is how to to allow information to be passed from the per-Exposure method to the per-Source methods - we should not do that via plugin instance attributes, which means we probably want to have the per-Exposure method return an arbitrary object that would be passed to the per-Source methods. Unfortunately, I don't see a way to avoid having that change the signature for those methods for all existing plugins, so this should be done relatively early, before we have too many user-contributed plugins.",3 DM-1249,"Reimplement CSS using JSON-packing, czar kazoo + c++ snapshotting phase 1","Transient cache for CSS, populated from python, no C++ interface to zookeeper. Pack multiple keys using json. Tenative packing spec is in DM-705 The v1 implementation transitions the zk access from the c++ layer into python, with code that understands how to unpack json-encoded data. This can coexist with the existing qserv_admin zk schema. v2 applies packing/unpacking logic in the creation/manipulation code in qserv_admin. ",10 DM-1250,"CSS design for query metadata v2",NULL,3 DM-1251,"CSS design for query metadata v1","The goal of this ticket (and DM-1250) is to try to understand what kind of per-query metadata is necessary to provide client-transparent query processing in case when czar/proxy could die or be restarted. Some relevant info is in the Trac: https://dev.lsstcorp.org/trac/wiki/db/Qserv/CSS/RunTimeState",3 DM-1252,"Implement structure for DB/table metadata in CSS",NULL,4 DM-1253,"Requirements gathering for Metadata Store",NULL,10 DM-1254,"Metadata Store - design v1","Research potential off-the-shelf candidates. Propose initial version of metadata design. ",5 DM-1255,"Metadata Store - experimental prototype v1","This first prototype will involve capturing metadata from a small set of image files. * Create a new schema file in the cat package (lsstSchema4mysqlW15.sql) with definition of tables that will be used to capture image metadata (see Exposure tables in other schema files), and some basic structure for capturing information about the entire repository. Decide which keywords are standard, and which should go to flat key-value area * Pick a directory (or several) with images, candidates: /lsst7/dr7/runs, /lsst7/releaseW13EP, /lsst3/DC3/data/obs/ImSim/pt1_2, /lsst/DC3/data/pt1.2.3k/datarel/ImSim * create test database, load schema * extract metadata for all images in a given directory and load metadata from fits headers.",10 DM-1256,"Metadata Store - design v2",NULL,5 DM-1257,"Metadata Store - experimental prototype v2 (DataCat)","Integrate prototype v1 with Fermi's DataCat (e.g., reuse logic for reading headers using afw, store in DataCat). Experiment with foreign tables.",6 DM-1258,"Update documentation and automatic install script w.r.t. Qserv 2014_09.0 release","Creation of qserv_distrib and distribution of Qserv via official LSST repositories have to be taken into account in Qserv documentation and automatic install script.",4 DM-1259,"evaluation of Cinder as OpenStack storage cache in the LSST Middleware","Deliverable: evaluation of Cinder as OpenStack storage cache in the LSST Middleware Daues G 100%"" ",100 DM-1260,"implementation of data movement in the production system in the existing hierarchy of NFS disk, condo, tape","Deliverable: implementation of data movement in the production system in the existing hierarchy of NFS disk, condo, tape Freemon M 50% ",54 DM-1261,"incorporation of sizing model into data center requirements","Deliverable: incorporation of sizing model into data center requirements Freemon M 100% ",7 DM-1262,"data center requirements document","Deliverable: data center requirements document Petravick D 10% ",11 DM-1263,"completed governance of security plan for review","Deliverable: completed governance of security plan for review Petravick D 10% ",1 DM-1264,"security plan october.","Deliverable: security plan Ephibian 10% ",1 DM-1265,"addition of procurement of physical goods to contract","Deliverable: addition of procurement of physical goods to contract Petravick D 5%, Gelman M 10% ",6 DM-1266,"insertion of wide area simulator into test stand","Deliverable: insertion of wide area simulator into test stand Freemon M 100% ",18 DM-1267,"report of traffic shaping","Deliverable: report of traffic shaping Freemon M 100% ",7 DM-1268,"upgraded KVMs","Deliverable: upgraded KVMs Mather B 50% ",9 DM-1269,"sizing model critique report","Deliverable: sizing model critique report Perez A 50% ",29 DM-1270,"more efficient VMware infrastructure","Deliverable: more efficient VMware infrastructure W15: Glick B 50%, Elliott M 25%, Mather B 10% 38 SP estimated S15: Mather B 40%, Glick B 25% 6 SP estimated",6 DM-1271,"technical roadmap critique report","Deliverable: technical roadmap critique report Perez A 50%",29 DM-1272,"Upgrade NFS Storage Servers with new hardware","Deliverable: new hardware, ZFS filesystem, 3x 100+ TB + 1 spare 100+ TB Elliot M 75% ",55 DM-1273,"deployment plan for version 1 of OpenStack","Deliverable: deployment plan for version 1 of OpenStack Glick B 75%, Elliot M 15%, Mather B 10%, Wefel P 10%",56 DM-1274,"Procure replacement development infrastructure","Procure, install, test, and deploy hardware to replace existing LSST development cluster infrastructure. Assignees: Bill Glick, Matt Elliot, Bruce Mather, Paul Wefel, Jason Alt Duration: November - December 2015",45 DM-1275,"level of effort","Deliverable: level of effort Wefel P 100%, Freemon M 5%",100 DM-1276,"level of effort","Deliverable: level of effort Voiciu L 100%",28 DM-1277,"level of effort","Deliverable: level of effort Petravick D 50%, Gelman M 50%, Glick B 30%, Mather B 50%",100 DM-1278,"written plan for next period epics","Deliverable: written plan for next period epics Petravick D 50%, Gelman M 50%, Glick B 50%",10 DM-1279,"Fix Scisql deployment test error (doc.py)","Deployment test in tools/docs.py fails due to a wrong ""scisql_index"" path in scisql documentation. Fortunately, qserv-configure.py doesn't stop on this error.",2 DM-1280,"meas_base ResultMappers should be FunctorKeys","The ResultMapper classes in meas_base should inherit from FunctorKey, and support bidirectional transfers involving the Result structs and records.",3 DM-1281,"add Schema method to join strings using the appropriate delimiter","Delimiters in Schema field names are version-dependent. One can currently use {{schema[""a""][""b""].getPrefix()}} to join fields using the appropriate delimiter, but this is confusing to read.",1 DM-1282,"multi-level replacement in Schema aliases","Schema aliases should support more than one level (i.e. an alias may resolve to another alias).",2 DM-1283,"remove meas_extensions_multiShapelet from release packages/buildbot",NULL,1 DM-1284,"rename meas_multifit to meas_modelfit, and add to lsst_apps",NULL,1 DM-1285,"Improve Startup of HTCondor Jobs","Adjust configuration parameters of HTCondor config and/or submission files to improve speed at which HTCondor jobs start in both the replicator pool and worker pool.",2 DM-1286,"Improve worker fault tolerance of missing distributor data","Instances of worker jobs might ask the Archive DMCS which distributor to connect to, only to connect to that distributor which has recently been rebooted, so the distributor might not have that information. At this point, the Distributor could send an “expired” notice of some kind to the Archive DMCS to clear it’s cache, and also tell the worker that it has no file of the type it’s looking for. The worker would then go back to the Archive DMCS. ",6 DM-1287,"Propose and document a recipe to build Qserv in eups","In-place build is available and documented.",2 DM-1291,"Identify test data and camera","To test processCcd we need to identify a set of data to process (possibly mocked). This implies that there will also need to be a minimal camera to go with the minimal data. This task is to identify the minimal data and find a location for it.",4 DM-1293,"Implement designed tests for processCcd","Implement the designed tests with the installed data.",8 DM-1294,"Generate use case","We should first sit down with a Camera team rep (Jim C. maybe) to define the tool they need. How dynamic are the data? What is the layout of the data? We should also get some example test data to work with.",6 DM-1295,"Sync test data headers with standards","The headers will need to be vetted against known FITS standards. The headers should be valid and should not contain any non-standard keywords. There should also be a census of FITS standards that may be used for defining sensor layout.",8 DM-1296,"Implement a tool to generate a Camera from FITS images","Implement the tool to generate the camera object. It should be a command line tool that will take a path to a file. It should have an option to persist the generated camera. It should also have the ability to plot the camera.",10 DM-1297,"Verify use of the tool with the camera team","We will need to run the tool in the camera team's system to make sure the interfaces are as they expected. It will also help to make sure that the tool addresses all of the cases.",6 DM-1298,"Design the API","The API will have to be able to accommodate data we know about, so will need to deal with reasonable missing data. It should also not preclude extension. This will need to be RFC'd since it is an API change.",8 DM-1299,"Implement and test the new API","Once the API is designed and signed off on, the API will need to be implemented and tested. This will require updating all obs_* packages that use the current interface.",10 DM-1302,"C++ code changes required for --std=c++11","Some C++ code requires changes for modern C++ compilers if it is to be compatible with C++11 and the older standard. Here is my list, so far (ignoring the known issue of MacOS having two different standard C++ libraries) Implicit conversion of shared_ptr to bool no longer works: In C++11, shared_ptr has an explicit operator bool which means that a shared_ptr can't be implicitly converted to a bool. This is likely to show up in a lot of packages. So far found in: - pex_policy - meas_algorithms warning: adding 'int' to a string does not append to the string seen in daf_persistence warning: 'va_start' has undefined behavior with reference types seen in pex_logging Trace.h and one other place. Fixed by not using references (and thus copying the arguments), rather than using pointers, to avoid changing the APIs. this is an old issue; but not all compilers warned about it: See warning: 'register' storage class specifier is deprecated seen in: - boost 1.55.0.1 - Eigen 3.2.0 - a pex_logging .i file I silenced this warning in sconsUtils because it's too much trouble to upgrade boost and Eigen, and the warnings are very obtrusive in daf_persistence: {code} src/DbStorageImpl.cc:87:35: warning: first declaration of static data member specialization of 'mysqlType' outside namespace 'persistence' is a C++11 extension [-Wc++11-extensions] dafPer::IntegerTypeTraits<1>::mysqlType = MYSQL_TYPE_TINY; {code} I have not figured out how to fix this one. clang 6 is pickier about the order of instantiation: in afw (fixed in DM-1302) Many errors such as the following: {code} include/lsst/afw/table/Flag.h:27:8: error: explicit specialization of 'lsst::afw::table::FieldBase' after instantiation struct FieldBase { ^~~~~~~~~~~~~~~ include/lsst/afw/table/Key.h:49:39: note: implicit instantiation first required here {code} Fixed by including Flag.h in Key.h clang 6 warns about ambiguous unsequenced modifications. Seen in shapelet, for example: {code} src/HermiteTransformMatrix.cc:107:59: warning: multiple unsequenced modifications to 'kn' [-Wunsequenced] for (int kn=jn, koff=joff; kn <= order; (koff += (++kn)) += (++kn)) { ^ ~~ {code} Fixed by using a += ++b, a += ++b, since it was the cleanest solution I could find. clang 6 is pickier about instantiation classes in the wrong namespace: in shapelet (fixed by not instantiating the anonymous classes): {code} src/MatrixBuilder.cc:945:1: error: explicit instantiation of 'lsst::shapelet::::SimpleImpl' must occur in namespace '' INSTANTIATE(float); {code} meas_algorithms produced this warning because the class member _wcsPtr was a reference (which was unecessary): {code} src/ShapeletKernel.cc:188:34: warning: binding reference member '_wcsPtr' to a temporary value [-Wdangling-field] _interp(interp), _wcsPtr(wcsPtr->clone()) ^~~~~~~~~~~~~~~ {code} clang 6 warns about expressions such as if (!a == 0) because the ! is applied to ""a"", not the result of the ""a == 0"". Fixed in the obvious way in several places. meas_algorithms seg-faults on loading unless boost is built with C++11 support. See ticket DM-1361 ",4 DM-1305,"Tests fail in shapelet when building on OS X 10.9","When building the master on pugsley.ncsa.illinois.edu, shapelet builds successfully, but two tests fail: {code} pugsley:lsstsw mjuric$ cat build/shapelet/tests/.tests/*.failed tests/testMatrixBuilder.py .F..... ====================================================================== FAIL: testConvolvedCompoundMatrixBuilder (__main__.MatrixBuilderTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/testMatrixBuilder.py"", line 310, in testConvolvedCompoundMatrixBuilder self.assertClose(numpy.dot(matrix1D, coefficients), checkVector, rtol=1E-14) File ""/Users/mjuric/test/lsstsw/stack/DarwinX86/utils/9.2+8/python/lsst/utils/tests.py"", line 328, in assertClose testCase.assertFalse(failed, msg=""\n"".join(msg)) AssertionError: 1/50 elements differ with rtol=1e-14, atol=2.22044604925e-16 0.175869366369 != 0.175869366369 (diff=1.99840144433e-15/0.175869366369=1.13629876856e-14) ---------------------------------------------------------------------- Ran 7 tests in 0.323s FAILED (failures=1) tests/testMultiShapelet.py ...F... ====================================================================== FAIL: testConvolveGaussians (__main__.MultiShapeletTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/testMultiShapelet.py"", line 88, in testConvolveGaussians self.compareMultiShapeletFunctions(msf3a, msf3b) File ""/Users/mjuric/test/lsstsw/build/shapelet/python/lsst/shapelet/tests.py"", line 107, in compareMultiShapeletFunctions self.compareShapeletFunctions(sa, sb, rtolEllipse=rtolEllipse, rtolCoeff=rtolCoeff) File ""/Users/mjuric/test/lsstsw/build/shapelet/python/lsst/shapelet/tests.py"", line 86, in compareShapeletFunctions rtol=rtolEllipse) File ""/Users/mjuric/test/lsstsw/stack/DarwinX86/utils/9.2+8/python/lsst/utils/tests.py"", line 328, in assertClose testCase.assertFalse(failed, msg=""\n"".join(msg)) AssertionError: 1/5 elements differ with rtol=1e-14, atol=2.22044604925e-16 2.44929359829e-16 != 1.13310777953e-15 (diff=8.881784197e-16/1.13310777953e-15=0.783842839795) ---------------------------------------------------------------------- Ran 7 tests in 0.131s FAILED (failures=1) {code} =============== More info on pugsley.ncsa.illinois.edu: pugsley:lsstsw mjuric$ sw_vers ProductName: Mac OS X ProductVersion: 10.9.5 BuildVersion: 13F34 pugsley:lsstsw mjuric$ clang -v Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn) Target: x86_64-apple-darwin13.4.0 Thread model: posix ============ The files are in {{/Users/mjuric/test/lsstsw/build/shapelet/}}.",1 DM-1306,"Pre-CCB review of LSE-140 docgen",NULL,2 DM-1309,"Edit agreed-upon changes into Word version of LSE-69","A meeting around 9/26/2014 agreed on a set of revisions to LSE-69, with some language still needed from [~gpdf]. This action is to edit the tracked-changes Word version of LSE-69 containing the notes from that meeting into a final copy that can be reviewed by the Camera team and used as input to editing the SysML version of the ICD.",3 DM-1310,"Create change request for LSE-69","Create a change request to bring LSE-69 up to date and capture the Summer 2014 work.",1 DM-1311,"Enter LSE-69 update into EA as SysML","Covers entering the contents of the LSE-69 update into EA as SysML, with associated updating of diagrams, and the creation of a docgen'ed version for CCB action.",1 DM-1312,"Proofread docgen'ed version of LSE-72","Brian Selvy is producing a SysML version of the LSE-72 updated edited by [~gpdf]. The action here is to proofread the docgen of that version once it is ready.",2 DM-1313,"Identify Conditions information in LSE-130 that is required for Alert Production","LSE-69 declares that there are two categories of Conditions data (telemetry) required by DM from the Camera: those items that are needed for Alert Production (for which the AP components at the Base will need a whitelist, and for which the Camera has a tighter latency requirement), and those that are not (but are then presumably needed in DRP or other deferred productions). It states that the subset needed for AP should be enumerated in LSE-130. The action here is to create an initial version of that list.",2 DM-1314,"Publish Qserv S14 version on lsst distribution server","In order to publish this version please tag Qserv master tip with ""2014_09.0"" and then run: {code:bash} ssh lsstsw@lsst-dev # command below can't be runned in buildbot, as it doesn't support qserv_distrib build rebuild -t 2014_09.0 qserv_distrib # bXXX is provided by previous command publish -t qserv -b bXXX qserv_distrib publish -t 2014_09 -b bXXX qserv_distrib {code} ",1 DM-1315,"advance to assigning tier-2 and tier-3 reliability levels ","Accommodated Ron Lambert's input on networking equipment. Assigned tier-2 and tier-3 levels to processing systems. ",4 DM-1316,"Deploy LSST stack within OpenStack instances on ISL testbed","Deploy the LSST Stack within OpenStack instances within the ISL testbed -- this could be for multiple flavors CentOS, Ubuntu, etc, and this could be done by pulling Docker Images to the instances. There will also likely be some initial debugging of starting instances within the ISL platform as a new installation has been stood up Sept 2014. ",1 DM-1317,"Create Docker Image / Dockerfile for LSST Stack for ubuntu","Create an installation of the LSST Stack v9_2 within a Docker Image for ubuntu for easing the import of LSST software into an OpenStack instance, We create the image utilizing a Dockerfile to make systematic the creation of such images. ",2 DM-1318,"update expected results file in SDSS demo test","Update the expected outputs in the SDSS DM stack demo repo to match what we expect from the new meas_base framework.",1 DM-1319,"Prepare quotes to order new VMware hardware & licenses",NULL,2 DM-1320,"Setup temporary NFS datastore for VMs","We expect the primary datastores will be local to the VM compute node, but this NFS datastore will allow us to live migrate a handful of VMs between compute nodes. We may also use this or another datastore as place to create backup snapshots of particular VMs.",2 DM-1321,"Setup new VM compute nodes","Once the new VM compute nodes arrive, the hardware needs to be installed, ESXi installed, and networked for use by vSphere.",6 DM-1322,"Expire Workers that receive no files","In instances where the files the worker expected to get never arrive, there should be a way of the worker to recognize this (say, a timeout after waiting for the Archive DMCS for some time), and exit.",4 DM-1323,"Configure and test new VM infrastructure","This would be configuring to work vSphere, setting up permissions and groups, adding datastores, and testing the use of the new setup.",10 DM-1326,"Automatic expiration of replicator jobs","Replicator jobs that receive no data specifically for the visit, raft, and exposure sequence ID within a certain amount of time should self expire to prevent future jobs from running. If this is not done, jobs will back up in the HTCondor queue.",4 DM-1328,"Order new NFS servers","We are waiting on the UIUC Business Offices to provide us a new account number for ordering these servers. As soon as we get that account number we need to order the hardware from KOI Computing.",6 DM-1329,"Plan new NFS mounts & data organization","This is a task of deciding the new client mounts for the 3 new NFS servers. These should match up with the new storage hierarchy and classification .",1 DM-1330,"Install new NFS servers",NULL,10 DM-1331,"squash edge errors in SdssCentroid","SdssCentroid doesn't trap exceptions that are thrown due to being too close to the edge, resulting in noisy warnings in the logs. Instead, it should catch the low-level exception and re-throw as MeasurementError, after defining a flag field for this specific failure mode.",1 DM-1332,"address no-shape warnings in GaussianFlux","GaussianFlux relies on the shape slot, and puts noisy warnings in the logs when the shape slot fails. However, we probably don't want to add a new flag for GaussianFlux to indicate this failure mode, because it'd be entirely redundant with the shape slot flag. We should figure out some other way to squash this warning - how we do that may depend on whether this is addressed before or after the C++ redesign. We should also consider having GaussianFlux add an alias to the schema to point back at the shape slot flag, creating what looks like a specific flag for this failure while actually just being a link back to the shape slot flag. That's probably not worth doing within the current C++ interface, however, as it'd require some unpleasant mucking around with ResultMappers.",2 DM-1333,"resolve factor of two difference in GaussianFlux","After changing the implementation of GaussianFlux to use the shape slot rather than estimate the shape itself by re-running the SdssShape code, Perry saw a 5-15% difference in the fluxes (I'm not sure of the sign). The new behavior (using the shape) is consistent with what we'd have gotten with the old code when the little-used ""fixed"" config option was enabled (not surprising, as that just looked up the SdssShape measurement by name, instead of via slots). I suspect the difference is coming in because of the factor of two between SdssShape's ""raw"" measurements - the actual Gaussian-weighted moments - and the factor of 2 it applies to make its measurements equivalent to ideal unweighted moments. The correct weight function to use for GaussianFlux includes this factor of 2 (i.e. it's larger than the ""raw"" moments), and it's likely either the old code wasn't including this or the new code isn't. We need to determine which one, and if necessary, fix the new code.",2 DM-1334,"Test the creation of basic OpenStack instances on the new ISL testbed [IceHouse]","A new version & implementation of the ISL OpenStack testbed is up and running. The new cloud is using IceHouse, the ninth OpenStack release. We get started on this platform by verifying that basic instance creation is working. We target the creation of an instance through the (Horizon) GUI interface, and via the nova CLI. ",1 DM-1335,"Create instance with a Floating IP Associated through the nova CLI","We see that in working with the Horizon GUI, it is fairly straightforward to give an instance a public IP address by associating a Floating IP with the current local IP. However, we will want to be able to accomplish this task both remotely and programmatically within workflow. As a step towards this, we target the solution of this via the nova CLI.",2 DM-1336,"Review advance doc to review with the data center working group","Consult with M. Freemon on the sizing model given the generic description of the computing needed tier-3 and tier-2 support. Make a pass through the underlying data center standards doc for things ""any competent designer should know"" for example wall plugs on separate circuits form computer circuits. add that to the document, and call meeting of working group for review in late Oct.",4 DM-1337,"Security Plan advancement for October","get a draft Project Office detail plan consistent with the level of devleopment of the master plan. (ephiphian). Draft a data classification plan, refer to plan and classes of data in drat materials. Obtain some central place in LSST documentation framework to hold materials being drafted (if supported by the LSST system) Recieve feedback based on portion of plan submitted to LSST project office. Incorporate feedback. (if minor) re-plan (if major)",18 DM-1338,"Make QSERV_RUN_DIR scripts able to detect qserv install paths using eups","Ticket related to Mario email (subject: [QSERV-L] Some points/actions from the discussion today) : Making your ""qserv data"" directory independent of where qserv is installed I think this is a big one, and largely independent of EUPS. You have a problem where you want to use one set of test data potentially with different qserv binaries (not at the same time, of course). I'd argue you should refactor the scripts generated by qserv-configure to either: * get _all_ their information about various paths from a _single_ file, for example, from etc/paths.cfg.sh. Then you can easily regenerate just that file when you need to switch to a different qserv (or zookeper, or what not), or... * refactor the generated scripts to learn from the environment which binaries to run. I.e., if $QSERV_DIR is defined, use that qserv, etc. This will let you switch binaries by simply setup-ing the new one with EUPS. The two are not mutually exclusive -- e.g., all of this logic could be in etc/paths.cfg.sh, and depending on whether this is a development build or a ""non-EUPS"" build, it can either pick up the paths from the environment or hardcode them. Assuming you did that, your development loop may look something like this: {code} # assuming that qserv-configure.py has already been run # in ../qserv-run # do something with qserv-a clone cd qserv-a setup -r . ... do some edits ... scons # now do the tests cd ../qserv-run ./bin/qserv-start.sh ... do tests ... ./bin/qserv-stop.sh # now switch to qserv-b clone cd ../qserv-b setup -r . # and do the tests again cd ../qserv-run ./bin/qserv-start.sh ... do tests ... ./bin/qserv-stop.sh {code} that is, as qserv-start picks up the relevant products from the environment, there's no need to rebuild/reconfigure the qserv-rundirectory each time. ",7 DM-1340,"Read through log4cxx documentation and log.git code","Read through the log4cxx documentation and become familiar with how the log.git package is set up.",2 DM-1341,"Write/configure tests with existing configurations and appenders","In order to be come more familiar with how to use the log.git package, write some tests to see how existing configurations and appenders are used by the log.git package.",4 DM-1342,"Write DM message appender class","Write DM message appender class to be used with log.git package. This might entail writing a configurator class as well; that depends on the investigation of how configurations/appenders are used.",16 DM-1343,"Write Unit test for new DM message appender class","Write unit tests for DM message appender class. This might also require some tests for a configurator class, if that class is created.",2 DM-1344,"Define gather/scatter mechanism for tasks with generic API","Define gather/scatter mechanism for tasks with generic API. This will use event services (DM Messages) initially, but we would like to support MPI or other communication mechanisms as well. This will involve consulting with Paul about how the existing code is structured. ",8 DM-1345,"OCS Middleware workshop","Attend OCS Middleware workshop. This will probably have be done remotely because I have a personal conflict with that time that will prevent me from attending in person.",4 DM-1346,"Write example programs for OCS Middleware","Write some example programs to get familiar with the OCS middleware software. The OCS middleware will later be integrated with the AP, in the base dmcs and replicator jobs.",4 DM-1347,"Refine Event base class to allow ActiveMQ filterable settings","The current Event.cc base class needs to be refined to remove and old-style data release terms that aren't used anymore. Plus, it needs to be easily extensible to allow other types of dictionaries of terms that will be used in the message headers to make them filterable on the server side.",2 DM-1348,"Update tests to use unit test framework","The tests for this package predate the unit test framework that other package use. Update the tests to uses the unit test framework and get rid of any duplicate or obsolete tests.",5 DM-1349,"Change marshaling code to use json","The marshaling code for non-standard (i.e., non-filterable) components of messages is custom and not standard. Change this to use JSON.",10 DM-1350,"General cleanup of Events package","There are some obsolete classes and code in the ctrl_events package that needs to be removed and/or updated. PipelineLogEvent, for example. That not only is no longer used, but it is applications specific, and should have been part of another package in the first place, subclassed from this package.",35 DM-1352,"ORIGINATORID value can churn too quickly.","The ORIGINATORID is a 64-bit word consisting of an IPv4 host address, 16-bit process id, and 16-bit local value. In addition to the 16-bit process id not being standard across platforms (Mac OS >Leopard goes to 99999), the churn rate for the local value should be much higher than just 16 bits. This could be fixed to changing ORIGINATORID to a 32-bit process id and a separate value for the local value, which would be specified together in the DM event selector. I have to look into this more to see if this is a viable solution. This might need to go to three separate values to future proof it (i.e., ipv6).",8 DM-1354,"Install docker 1.1.2 in an ISL OpenStack CentOS instance, perform basic checks","Install docker 1.1.2 in an ISL OpenStack CentOS 6.5 instance, and perform basic checks such stopping and starting the docker daemon, changing default settings such as size limit of containers/images, pulling standard images from docker hub, starting containers from these images, etc. ",2 DM-1355,"Re-arrange how Qserv directories are installed","we already touched question about installation directory structure at a meeting, maybe we can improve things by re-arranging how things are installed we are currently installing stuff into four directories: cfg, bin, lib, proxy to make it look more standard and to avoid clash with qserv source directories we could move cfg and proxy to a different location (something like share/qserv to make it more root-install-friendly in case we ever want to install under /usr) this change (if you want to do it) deserves separate ticket, do not do it in this ticket ",3 DM-1358,"End-to-end demo fails to exit with the correct status when Warning would be correct.","The output of demo2012 results in an output file which is compared against a benchmarked file. Currently the comparison allows a deviation from the benchmark based ""on the the number of digits in the significands used in multiple precision arithmetic""; that number is currently set to 11. An example using that setting is: @ Absolute error = 5.9973406400e-1, Relative error = 9.1865836000e-4 ##2566 #:25 <== 29.7751550737 ##2566 #:25 ==> 29.7478269835 Additionally, the current use returns: a 'Fatal' error if the code itself fails to execute correctly; a ""Warning' error if the any of the benchmarked quantities do not meet the comparison criteria; 'Success' if the comparison meets all criteria. It is noted that the buildbot 'Warning' color indicator is not currently being displayed when the comparison fails. That is a coding error. This Issue will: * ensure that BUILDBOT_WARNING(S) causes the correct display color when appropriate. This Ticket has been split into two parts. This is part 1. Part 2 is DM-1379 ",2 DM-1359,"improvements to PsfFlux","While using it as an example for the redesign of DM-829, I came up with some ideas for how to improve PsfFlux's handling of edge/bad pixels: - Add a flag indicating that at least one pixel was rejected - Add thresholds (number? fraction of PSF model size?) for how many non-rejected, non-edge pixels we need to even attempt a fit.",1 DM-1360,"Fix minor loose ends from new result plumbing","Some of the more minor issues raised in comments from DM-199 were left undone prior to merging. This ticket addresses them. For more information, please see: https://github.com/LSST/qserv/pull/2 . ",1 DM-1362,"Edit pull interface and other Summer 2014 work into LSE-68 in Word","Deliverable: circulate a Word-based draft of LSE-68 in which the ""push"" interface is removed, and the ""pull"" interface is refined to include the guider and other Summer 2014 work. Note that the use of ""pull"" for the guider applies whether or not the proposed guider redesign is accepted.",2 DM-1363,"Avoid use of ~/.my.cnf (used by css watcher)","See https://jira.lsstcorp.org/browse/DM-1258?focusedCommentId=29230&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-29230. {{~/.my.cnf}} is used by css watcher, an optional tool used for monitoring css. It is a symlink to $QSERV_RUN_DIR/etc/my-client.cnf. The css watcher could use MySQL credentials located in ~/.lsst/qserv.conf (used by integration tests wich are a Qserv/MySQL client)",2 DM-1364,"replace ""bad data"" flag in SdssCentroid","SdssCentroid has a ""bad data"" flag that doesn't actually convey any information about what went wrong. This should be replaced with one or more flags that provide more information.",3 DM-1372,"Errors in testHealpixSkyMap.py","There is a failing unit test when healpy is supplied. The problem is that the method boundary is not defined in the version of healpy we supply for the sims, however boundaries does exist. If I replace boundary with boundaries, the test passes.",1 DM-1375,"Create Docker Image / Dockerfile for LSST Stack for CentOS6.5","Make a Dockerfile for systematic generation of docker images using a Centos6.5 base image containing the LSST Stack (v9_2 at the moment) and library dependencies.",2 DM-1376,"Ensure that the partition package is C++11 clean and compiles on OSX 10.9","The LSST buildbot infrastructure recently changed to building everything with --std=c++0x, which broke the partition package, and hence automated Qserv builds. While debugging this, I discovered that the partition package does not build on OSX 10.9, and considering how minimal its dependencies are, it really should. The OSX issue can be fixed by avoiding {{using boost::make_shared}}. The partition package should be cleaned up to avoid all use of {{using}}. If we decide to use C++11 in Qserv, then the codebase should also be modernized (in particular, there are use-cases for static_assert, nullptr, etc... ). ",2 DM-1377,"Develop expanded and updated Information Security Program from NSF guidance/templates, using existing documents (e.g. LSE-99) as starting point ","Develop expanded and updated Information Security Program from NSF guidance/templates, using existing documents (e.g. LSE-99) as starting point. Create master document and subordinate documents, allowing for existing AURA, site, and institutional polices to be incorporated as appropriate. JK: In PMCS this is Petravick D and Ephibian (LeClair L)",56 DM-1383,"Investigate deblending in one band followed by using the resulting templates in all bands","(See also [HSC-1025| https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1025]) One poor-man's approach to deblending multiple bands and visits is - Deblend in one band (or a combination, e.g. chi^2 band) using an SDSS-like algorithm that produces templates for each child - Take the templates (or possible model fits to those templates) and use them to deblend the objects in all other bands It will probably be necessary to include undetected objects in this fit (cf. [HSC-1023|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1023]), using point-source models. Note that this does not allow for the different seeing in each band. This is not obviously catastrophic for reasonably wide blends as the total flux in each pixel is conserved, and if only one template is important near an object's centre the exact form of the template is unimportant.",40 DM-1384,"Investigate deblending in one band followed by linear fits of models","(Cf. [HSC-1024|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1024]) One poor-man's approach to deblending multiple bands and visits is - Deblend in one band (or a combination, e.g. chi^2 band) - Fit models in that band - In all bands separately fit those models simultaneously to all object in the blend, allowing only a minimum of coefficients to float (ideally only amplitudes, but given the realities of real galaxies the bulge and disk will need to both be fit, or the Sloan Swindle components, or the Lensfit Swindle, or ...). The correct PSF for each image would be used. It will probably be necessary to include undetected objects in this fit (cf. [HSC-1023|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1023]), using point-source models. It is not clear how much of a poor-man's solution this actually is, or whether it will work quite well.",40 DM-1386,"Merge Footprints from different bands/epochs","The current concept of the deblender assumes that the inputs are - A merged set of Footprints that define which pixels are part of the blend - A merged set of Peaks within that merged Footprint Please generate these merged Footprints (which will be defined in (x, y) coordinates in the tract/patch coordinate system). ",5 DM-1387,"Generate a master list of Objects given detections in multiple bands/epochs","Once we've detected Sources in multiple bands we need to merge the positions to generate Objects. This is a little complicated (or at least messy): - The positions have errors - If the seeing is different in different visits, objects may be blended in some but not all exposures - If we use more than one detection pass (e.g. smoothing when looking for faint objects, not smoothing for bright) this has similar-but-different consequences (but we should probably deal with in the per-band processing) - Objects move, so even if the positions are within the errors the motion may still be detectable ",5 DM-1388,"Submit LCR for LSE-68","Create an LCR, including a summary of changes, for LSE-68.",2 DM-1389,"Edit LSE-68 changes into EA",NULL,2 DM-1390,"Improve output from integration tests","Currently the integration tests print pages and pages of output, it is hard to see what failed and how. It'd be nice to clean that out, and instead, print some short summary, perhaps in a form of a summary table what queries has run, what succeeded, what failed etc. HTML format might be a reasonable way to display it.",6 DM-1394,"Eups 1.5.4 requires each new shell to source the eups setups.sh","eups v 1.5.4 requires each new shell to source ...eups/../bin/setups.sh. This requires the buildbot scripts: runManifestDemo.sh, create_xlinkdocs.sh, be updated to individually do that task. Add demo2012: bin/demo.sh . ",2 DM-1395,"Reimplement CSS using JSON-packing, czar kazoo + c++ snapshotting phase 2","The v1 implementation (DM-1249) transitions the zk access from the c++ layer into python, with code that understands how to unpack json-encoded data. This can coexist with the existing qserv_admin zk schema. v2 applies packing/unpacking logic in the creation/manipulation code in qserv_admin. v2 scope will be defined further, once DM-1249 enters review. Update: Phase 2 includes removal of c++ zk code and unification of Qserv css modules into a single place.",10 DM-1396,"Design CSS schema to support database deletion","Need to implement deleting databases. Deliverable: a design of the system that will be capable of deleting a distributed database including all copies of that database on all workers, all replicas of all chunks are deleted. It should be possible to ""create database x"" at any time later.",2 DM-1400,"Improve documentation of pixel systems in obs_lsstSim","There is not much documentation of the coordinate systems in use by CameraGeom. This is by nature documentation that is instrument specific, so should go in the obs_ package for each instrument.",1 DM-1404,"Create suite of Dockerfiles / docker images for LSST Stack for ubuntu, CentOS","Building on issues DM-1317 and DM-1375 where initial images and Dockerfile's were constructed, we can now use these Dockerfile's as prototypes to extend the set of Dockerfiles & images. We observe that by making simple (scriptable) edits to the initial Dockerfile, we can run 'docker build' to make docker images for several combinations of OS and base compiler gcc version.",2 DM-1405,"Prepare a SL6x openstack image for with Qserv development environment","Qserv packaging procedure requires to often rebuild Qserv and relaunch integration tests. IN2P3 Openstack platform offer next virtual machines : {code:bash} [fjammes@ccage030 ~]$ nova flavor-list {code} | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | | 1 | m1.tiny | 512 | 0 | 0 | | 1 | 1.0 | True | | 15 | cc.windows.small | 4096 | 20 | 0 | | 2 | 1.0 | True | | 16 | cc.windows.xlarge | 8192 | 50 | 0 | | 4 | 1.0 | True | | 2 | m1.small | 2048 | 10 | 20 | | 1 | 1.0 | True | | 3 | m1.medium | 4096 | 10 | 40 | | 2 | 1.0 | True | | 4 | m1.large | 8192 | 10 | 80 | | 4 | 1.0 | True | | 5 | m1.xlarge | 16384 | 10 | 160 | | 8 | 1.0 | True | | 6 | cc.lsst.medium | 4096 | 20 | 40 | | 2 | 1.0 | False | | 7 | cc.lsst.large | 16384 | 20 | 160 | | 8 | 1.0 | False | | 9 | cc.lsst.xlarge | 40000 | 20 | 160 | | 20 | 1.0 | False | cc.lsst.xlarge would allow a quick build/test of new Qserv release.",5 DM-1407,"Add return code for integration tests","Integration test have to returns non-zero when failing. This will ease use of CI or debugging tools ({{git bisect}}, buildbot).",1 DM-1411,"Stand up netem server with Dell PE 1950",NULL,4 DM-1424,"Create persistent volume of Cinder block storage and attach to instance","We create a persistent volume of Cinder block storage and attach to working instance. When it was created, the instance does have a specified amount of ephemeral disk, but this disk will be destroyed with the instance. We want to test that we can create a persistent volume of block storage, attach it to an instance, format the storage with a file system, and mount the volume for use with processing, where data/output can be retained after the instance is destroyed. ",1 DM-1426,"Document how to switch Qserv or dependency in eups","DM-1338 may allow to switch Qserv or dependency version using eups, without having to re-configure Qserv (i.e. without any change in QSERV_RUN_DIR). There may have conditions for this to work. That's why it's reasonable to get some user feedback on DM-1338 before documenting this feature.",2 DM-1428,"Add return codes for qserv-check-integration.py and qserv-testunit.py",NULL,1 DM-1429,"Improve Qserv Configuration Procedure","Based on input from review of DM-1338, and discussions at hangout https://confluence.lsstcorp.org/display/DM/LSST+Database+Hangout+2014-10-29: implement -keepdata option. * --keepdata will preserve qserv_meta.conf and my.cnf. Regarding the latter, this is necessary because user might have customized data_dir. * The structure of qserv_meta.conf and my.cnf must remain the same between existing version of Qserv and the to-be-configured version of Qserv. It is users's responsibility to ensure the structure did not change. Note in particular, the template files (with the exception of my.cnf) should not be customized",7 DM-1431,"Process sample sdss data with LSST Stack in a Docker container in OpenStack","Process sample sdss data, starting with the lsst_dm_stack_demo, with LSST Stack in a Docker container in an OpenStack instance. ",2 DM-1432,"SUI define local hardware needs to host the test DB and application servers","We need have Qserv and the test DB in IPAC to do development and test locally, to access the data through Qserv APIs.",10 DM-1433,"SUI install Qserv and test DB, enable access to catalogs for SUI team members","Install Qserv and test DB in local hosts. Enable SUI team members to access the catalogs. ",6 DM-1434,"State diagram for jobs in progress","Build a state diagram showing job progress throughout a run.",13 DM-1439,"afw tests use the same name for a dummy file: test.fits","The afw package uses a file named: test.fits in multiple testers. If the user sets up the build to use multiple CPU (-j #), then there is the risk that the shared filename will be affected by more than one tester at a time. In the case which provoked this Issue, the tester: testSimpleTable.py, reported that the file was missing. A simple rerun managed to get past the error. I recommend that the different testers use uniquely named demo files.",1 DM-1440,"Github Transition: Naming conventions for repositories","The Simulations team has requested that repos in general and the numerous DM repos in particular are prefixed in a way that would make fitering them out of searches and display easy (for example, their repos are prefixed sims_*) This would be an evident useability aid to DM developers and outside contributors too. Obtain a decision on how to allow users to quickly isolate repositories they are interested in. ",3 DM-1441,"Github Transition: Storage of large data files","Github currently has fixed 100MB per blob or 1GB per repo limits. Sims has at least one file in its repo whose history exceeds this (sims_meas) and this has been raised as an issue before. Obtain a decision on a suitable way forward for the time being that would allow the upload of the repositories on Github. Experienced to be reviewed in a few months time to consider whether has proved satsifactory.",4 DM-1442,"Github Transition: Stash-stored pull requests, extraction","Extract comments from Atlassian Stash made during pull requests /code reviews and their associated SHA1s if it all possible. ",4 DM-1443,"Github Transition: pull request discussion, retention - proof of concept","Store review comments / PR discussion into relevant git repositories Code & process to do this on a continuous basis post-transition will be a different issue, ",1 DM-1444,"Test absence of individual components","Test absence of individual components of the AP simulator. Bring each down, run the system, and restart just those components to see if the system still operates as expected",2 DM-1445,"check if Qserv compiles with C++11",NULL,1 DM-1446,"Fine tune czar and worker database initialization","For now, Qserv databases are the same on the master and on the worker. It means that czar tables are created on the worker and vice-versa. This ticket aims at creating the right tables at the right places, and also at sharpening permissions on these tables. This is done by splitting configuration script qserv-czar.sh, which configure a mono-node instance, in two scripts: - qserv-czar.sh, for configuring the czar - qserv-worker.sh, for configuring workers. see https://confluence.lsstcorp.org/display/DM/LSST+Database+Hangout+2014-10-29 Please note that this code can't be validated with mono-node integration tests (whereas it doesn't break them): indeed it requires a mono-node installation with 2 MySQL instance, one for the worker and one for the master. Updating the mono-node configuration and integration tests with such a feature would make them far more complex. A quick and dirty, hard-coded, testbed is available in u/fjammes/DM-1446-test. It has been used successfully to test this ticket. This ticket will also be validated during next Qserv install on in2p3 clusters.",4 DM-1447,"Improve spatial-selection flexibility by parsing ptInSphBox-like syntax instead of qserv_areaspec_box","Note: it is not clear that we should do this. This is an idea to change the syntax for spatial area selection. Currently, we have SELECT...FROM...WHERE qserv_areaspec_box(...) ... This forces the area selection to apply the box (or appropriate) cut on the tables referenced in the FROM list that are partitioned, with those columns. It would be nice to allow spatial selection based on tables (and columns?) explicitly specified by the user, and then compute the appropriate chunk coverage and WHERE clause conditions and predicates. This is a pretty big deal, and would affect all spatial-select user queries, though the compatibility can be mitigated by supporting both syntaxes.",100 DM-1448,"Move code for mock images into afw so it reusable.","There is some code in the exampleUtils in $IP_ISR_DIR/examples that could be of wider use. Specifically there is code to generate mock darks, flats, and raw data from a mock camera. There is also code to generate a mock dataRef. It could be used more widely if moved someplace else. Russell suggested afw.cameraGeom.utils.",1 DM-1449,"newinstall.sh should check that python2 is available","The standards now suggest using: {code} #!/usr/bin/env python2 {code} instead of {code} #!/usr/bin/env python {code} This doesn't work with (at least) Mac OSX 10.9 system python. newinstall.sh should check if this works and suggest a fix (creating a symlink).",1 DM-1452,"Detect lua version in admin/templates/configuration/etc/init.d/mysql-proxy",NULL,1 DM-1453,"Add data versions to Zookeeper","Track versions of data inside zookeeper, and detect from Qserv code if Qserv code is compatible with given format of data.",2 DM-1454,"LOE - Week ending 10/31/14",NULL,8 DM-1455,"LOE - Week ending 10/24/14",NULL,6 DM-1456,"finish porting meas_algorithms unit tests",NULL,4 DM-1461,"C++ Redesign -- Result definition for custom algorithms","Additions to Jim's redesign to make it easier to define custom results.",3 DM-1462,"Add NaN check to PixelFlags","The test of PixelFlags in measureSources.py (from measAlg) requires a check to be sure that the center inputs are not NaN.",1 DM-1463,"SdssShape shiftMax config item is being ignored","The code we ported from meas_algorithms sets the maxShift to 2 without regard to the config item which is supposed to set that value.",1 DM-1464,"Design Review prep for C++ redesign","Write up the design developed on DM-829 and push it through review.",1 DM-1469,"Github Transition Plan: Reverse mirror for beta-tester repositories","Test the reverse mirror for beta-testers. Straw man: 1. Break the mirror for anybody beta-testing github workflow 2. Mirror back to new gitolite area: ""mirror"") Method: https://help.github.com/articles/duplicating-a-repository/ ",1 DM-1471,"Rewrite obs_test","I want to use obs_test to test processCcd, but it contains no imaging data. I will replace the current data with new imaging data (a portion of one LsstSim CCD), add a camera and rewrite the mapper (subclassing CameraMapper). This also requires rewriting the unit tests in obs_test and in one other package that uses obs_test.",10 DM-1473,"Execute multi-platform lsst_dm_stack_demo test with Fig orchestration","In DM-1431 we demonstrated processing of sample data within Docker containers within an OpenStack instance. Data was processed for containers based on CentOS6.5, Ubuntu 13.10, Ubuntu 14.10. We consider a multi-platform ""testing"" scenario, where we process the sample data in numerous containers based on various platforms/OSs all simultaneously on an OpenStack instance. This scenario entails starting up and managing multiple Docker containers. Fig (http://www.fig.sh) is a new tool for starting up and managing services via Docker containers. It is a common use case with Docker to have individual components of an application each run separately in a container (resulting in numerous containers running on a node or cloud env, i.e., a 'Docker stack'), with the containers having linkages/dependencies to be managed. Fig may be used to encode the relationship between the containers in the Docker stack, and to start up such a set of containers. We install and examine Fig, and apply Fig to our multi-platform ""testing"" scenario. ",4 DM-1475,"Fix 2014_09 documentation","Replace {{NEWINSTALL_URL=http://sw.lsstcorp.org/pkgs/}} with: {{NEWINSTALL_URL=http://sw.lsstcorp.org/eupspkg/}} and {{eups distrib install qserv_distrib 2014_10.0}} with: {{eups distrib install qserv_distrib -t 2014_10}}",1 DM-1476,"Secure MySQL root password in configuration templates","MySQL password in written in multiple file during configuration procedure. One single file (QSERV_RUN_DIR/tmp/my.cnf) should be used, and removed at the end of configuration procedure. qserv-meta.conf also contains MySQL password and should be also secured (move password to qserv-configure.py cmd line?).",4 DM-1477,"Study fig to manage Qserv cluster","http://www.fig.sh/ coordinate with @GregDaues who is also investigating fig.",5 DM-1480,"Buildbot master takes exception when exiting from mail notifier after dynamic email sent.","Buildbot master exits without posting the required statically-addressed email notification if a dynamically-addressed was sent. This fix needs to ensure that the required (by buildbot specification) static email is sent even if it has to be directed to a dead-letter box (which it is).",3 DM-1481,"Discover/learn what others are doing in astronomy software","Attend the annual ADASS conference to keep up with the software development in the astronomy community. Trey Roby, Tatiana Goldina, Xiuqin Wu plan to attend the ADASS 24.",20 DM-1482,"SUI: study other plot packages","Firefly uses clientsidegchart at https://code.google.com/p/clientsidegchart/ for XY plot. This package has not been updated since 2010. We need to find out more about other plotting packages and consider if we need to switch. The candidates are: 1. flot plotting library for JQuery. http://www.flotcharts.org/ 2. Highcharts JS at http://www.highcharts.com/",10 DM-1488,"Finalize the Design of Query Metadata","Fine-tune the design outlined in DM-1251 (metadata for capturing information about running queries in Qserv.).",6 DM-1489,"Modify czar to interact with query metadata","Modify czar code: it should interact with the query metadata (store information for long running queries, retrieving).",6 DM-1490,"FY18 Design and Implement Query Cost Estimate","Design and implement system that will estimate query cost. In particular, we will need to know if the query is interactive, or should be scheduled on shared scan, which shared scan etc. Estimating cost will likely involve looking at number of chunks involved, number of joins, and complexity of math operations. Consider extending API and allow users to specify a hint. ",79 DM-1493,"Add queries on partitioned table for Qserv integration test dataset #3","Qserv test dataset #3 runs only queries on non-partitioned tables. Adding queries on partitioned table would increase drastically the coverage of these tests. Please note that a long-term solution could be to launch all the *.FIXME queries and to have a more detailed report. For example define queries which must pass for the integration test to succeed, and test queries which may pass and log their results in a report.",4 DM-1495,"Allow newinstall.sh to run in batch mode without installing Anaconda","Installing Qserv on a cluster without internet access requires newinstall.sh to run in batch mode without installing Anaconda (whose install requires internet access). So Anaconda should have a {{-anaconda=yes|no}}. Usefulness of git {{-git=yes|no}} option, against full batch option answering ""yes"" everywhere could also be studied.",2 DM-1497,"Package Qserv mono-node instance in Docker","Learn Docker basics and then package a Qserv mono-node instance.",5 DM-1498,"Package Qserv master and worker instance in Docker","Learn Docker basics and then package a Qserv mono-node instance.",5 DM-1505,"confusing error message when enabling unregistered items in RegistryField","pex_config seems to split out this confusing error message when trying to enable (i.e. append to .names) a registry item that doesn't exist: {code:hide-linenum} File ""/home/lam3/tigress/LSST/obs_subaru/config/processCcd.py"", line 51, in root.measurement.algorithms.names |= [""jacobian"", ""focalplane""] File ""/tigress/HSC/LSST/lsstsw/anaconda/lib/python2.7/_abcoll.py"", line 330, in __ior__ self.add(value) File ""/tigress/HSC/LSST/lsstsw/stack/Linux64/pex_config/9.0+26/python/lsst/pex/config/configChoiceField.py"", line 72, in add r = self.__getitem__(value, at=at) AttributeError: 'SelectionSet' object has no attribute '__getitem__' {code}",1 DM-1506,"Support new version of newinstall.sh","newinstall.sh now creates loadLSST.bash instead of loadLSST.sh. This has to be taken in account in Qserv automated install script: qserv-install.sh and in Qserv documentation.",1 DM-1508,"Reverse image slicing doesn't work as expected","The lsst.afw.image image-like classes support slicing, but reverse slicing does not work as expected. Here are some examples: {code} from lsst.afw.image import ImageF im = ImageF(10, 10) im[:,:].getDimensions() # is (10,10) as expected im[0:10, 0:10].getDimensions() # is (10,10) as expected im[::-1, ::-1] # should be the data with x and y reversed, but fails with: Box2I(Point2I(-2,-2),Extent2I(12,12)) doesn't fit in image 10x10 im[9:1:-1, :].getBBox() # starts at 0,0, as expected, but is 10x10 instead of 9x10 {code}",1 DM-1509,"Remove unnecessary config/install steps","Some existing installation configuration steps are not needed any more. For instance, we don't need $RUN/q or $RUN/result any more-- these paths are handled directly by qserv code, so the filesystem lookup shouldn't ever take place. Completion of this ticket should remove unnecessary steps and concepts from installation, simplifying future maintenance. ",2 DM-1510,"C++ Standards Rule 3-9 needs rewrite","Jim Bosch reviewed Section 3 of the C++ Standard. He noted this change which I felt warranted SAT clarification: 3-9: This is a confusing conflation of two entirely different concepts: it seems to discourage all global variables, regardless of whether they're in a namespace, but only to discourage free functions when they aren't in namespace. If that reading is correct, I think both are sensible recommendations, but they need to be clarified, and probably split up. If the reading should be that all free functions are discouraged, that'd be a terrible rule we violate all over the place. If the reading should be that global variables are only discouraged when not in a namespace, that needs to be clarified (and IMO it could be bumped up to a complete prohibition). Refer to: https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908685 Rule 3.9. The current Rule states: 3-9. Global variables and functions SHOULD be avoided and if used MUST always be referred to using the '::' operator. _________________________________________ ::mainWindow.open(), ::applicationContext.getName(), ::erf(1.0) __________________________________________ In general, the use of global variables should be avoided. Consider using singleton objects instead. Only use where required (i.e. reusing a framework that requires it.) See Rule 5-7 ( .https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908706#C++Statements-5-7 ). Global functions in the root namespace that are defined by standard libraries can often be avoided by using the C++ versions of the include files (e.g. ""#include "" instead of ""#include ""). Since the C++ include files place functions in the std namespace, ""using namespace std;"", which is permitted by Rule 5-41, will allow these functions to be called without using the '::' operator. In cases where functions are only available in the C include files, the '::' operator must be used to call them. This requirement is intended to highlight that these functions are in the root namespace and are different from class methods or other namespaced free functions. ",1 DM-1514,"calling extend with a SchemaMapper should support positional arguments","Calling {{catalog.extend(other, mapper)}} isn't equivalent to {{catalog.extend(other, mapper=mapper)}} because the second argument is the boolean {{deep}}. When a SchemaMapper is passed as the second argument, we should recognize it for what it is.",1 DM-1515,"sconsUtils fails to identify Ubuntu's gcc","On Ubuntu 12.04, gcc --version says: {code:hide-linenum} gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3 Copyright (C) 2011 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. {code} This apparently isn't quite what sconsUtils expected, because it says: {code:hide-linenum} scons: Reading SConscript files ... Checking who built the CC compiler...(cached) error: no result CC is unknown version unknown {code} Happily, everything seems to work anyway, as the fall-back options for the unknown compiler work fine with this one.",1 DM-1516,"Use eups swig instead of system swig during Qserv build","Qserv build system use system swig, it should be fixed to use eups swig. ",3 DM-1518,"Add support for ""SET @@session.autocommit""","The SUI team is using JDBC connector and queries like ""SET @@session.autocommit = {0}"".format(switch) are derailing everything. We should add support for these queries. ",2 DM-1519,"check out FIrefly package, build and run an applicaiton","Get familiar with FIrefly package, build and run an application. Understand the coordinate grid overlay on an image. ",10 DM-1520,"Fix confusing error message","Selecting from a table that does not exist, e.g. something like: select count(*) from whdfd produces a strange error: ERROR 4110 (Proxy): Qserv error: 'Unknown error setting QuerySession' ",1 DM-1521,"Fix ""stripes and substripes must be natural numbers"" bug","on user side: {code} ERROR 4110 (Proxy) at line 1: Qserv error: Unexpected error: (, ConfigError(), ) {code} in czar log: {code} 1114 22:30:10.155 [0x7fafcefab700] ERROR root (app.py:370) - Unexpected error: (, ConfigError(), ) Traceback (most recent call last): File ""/usr/local/home/becla/qservDev/Linux64/qserv/2014_09.0/lib/python/lsst/qserv/czar/app.py"", line 360, in __init__ self._prepareForExec() File ""/usr/local/home/becla/qservDev/Linux64/qserv/2014_09.0/lib/python/lsst/qserv/czar/app.py"", line 437, in _prepareForExec self._addChunks() File ""/usr/local/home/becla/qservDev/Linux64/qserv/2014_09.0/lib/python/lsst/qserv/czar/app.py"", line 535, in _addChunks self._computeConstraintsAsHints() File ""/usr/local/home/becla/qservDev/Linux64/qserv/2014_09.0/lib/python/lsst/qserv/czar/app.py"", line 529, in _computeConstraintsAsHints self.pmap = self._makePmap(self.dominantDb, self.dbStriping) File ""/usr/local/home/becla/qservDev/Linux64/qserv/2014_09.0/lib/python/lsst/qserv/czar/app.py"", line 478, in _makePmap raise lsst.qserv.czar.config.ConfigError(msg) ConfigError: ""Partitioner's stripes and substripes must be natural numbers."" {code} To reproduce: {code} qserv-check-integration.py --case-no 01 --load {code} then {code} mysql --port 4040 LSST -e ""select count(*) from Object"" {code} ",2 DM-1524,"Switch readMetadata' return value from PropertySet to PropertyList","It'd be useful if readMetadata would return PropertyList instead of PropertySet, because in many situations order matters. For details, see discussion in DM-1517",2 DM-1525,"Investigate halos around stars","Andrew Becker reports on dm-users: {quote} We have been using the LSST stack to reduce CFHT data at UW, and have come across a potential bug in the DM software. I don't see how this could be something intrinsic to the data, instead it seems like it could be a bug in the software triggered by its application to new data. I have a how to reproduce at NCSA at /lsst/home/becker/CFHT. Follow commands in the SETUP file to get an up-to-date version of the stack (b449) and a modified version of ip_isr (a hack to avoid overscan correction) and a private branch of obs_cfht (u/krughoff/ossos_support). If you run the commands in SOURCE_ME it will run the processCcd on a single CCD, in 2 modes, yielding 2 output directories. The second call merely sets the bin sizes to be the same amongst all the various background subtractions, but is sufficient to trigger the bug. A basic script included in the directory differences the 2 output calexps (a straight -=) and ds9 is used to view the diff. If you pan to e.g. 1845,546 you'll see a feature like attached in the difference (out1 on the left, out2 in the middle, diff on the right). This can be seen in various other parts of the image, as irregular annuli around bright blended objects. It suggests to me that during measurement some substamp manipulations are leaking into the original image. But you can kind of see in the center image that the second run seems to trigger the issue. {quote} And then: {quote} So to summarize: export LSSTSW=~lsstsw source ~lsstsw/bin/setup.sh setup lsst_apps -t b449 setup -k -r /nfs/lsst/home/becker/LSST/DMS/obs_cfht setup -k -r /nfs/lsst/home/becker/LSST/DMS/ip_isr setup -k -r /nfs/lsst/home/becker/LSST/DMS/processFile cd /nfs/lsst/home/becker/CFHT will set up your environment. The following is sufficient to recreate the bug, and create postIsrCCD images for processFile: processCcd.py input/ --id visit=1612606 ccd=8 --output out1/ --config isr.doAssembleDetrends=False isr.fringeAfterFlat=False isr.fringe.pedestal=False isr.doBias=False isr.doFlat=False isr.doFringe=False isr.doWrite=True calibrate.doPhotoCal=False calibrate.doAstrometry=False doDeblend=True processCcd.py input/ --id visit=1612606 ccd=8 --output out2/ --config isr.doAssembleDetrends=False isr.fringeAfterFlat=False isr.fringe.pedestal=False isr.doBias=False isr.doFlat=False isr.doFringe=False isr.doWrite=True calibrate.doPhotoCal=False calibrate.doAstrometry=False doDeblend=True detection.background.binSize=512 calibrate.detection.background.binSize=512 calibrate.background.binSize=512 Following are the calls I made to processFile.py (using my modified version of the processFile.git package), and which return images whose difference is exactly 0. So I don't really get whats happening with processFile, but it doesn't seem to be accepting my command-line config changes. processFile.py out1/postISRCCD/13AP06/E+0+0/2013-03-11/r/postISRCCD-1612606-08.fits --outputCalexp calexp1.fits -c calibrate.doPhotoCal=False calibrate.doAstrometry=False doDeblend=True calibrate.repair.doCosmicRay=False processFile.py out2/postISRCCD/13AP06/E+0+0/2013-03-11/r/postISRCCD-1612606-08.fits --outputCalexp calexp2.fits -c calibrate.doPhotoCal=False calibrate.doAstrometry=False doDeblend=True calibrate.repair.doCosmicRay=False detection.background.binSize=512 calibrate.detection.background.binSize=512 calibrate.background.binSize=512 processFile.py out2/postISRCCD/13AP06/E+0+0/2013-03-11/r/postISRCCD-1612606-08.fits --outputCalexp calexp3.fits -c calibrate.doPhotoCal=False calibrate.doAstrometry=False doDeblend=True calibrate.repair.doCosmicRay=False detection.background.binSize=1024 calibrate.detection.background.binSize=1024 calibrate.background.binSize=1024 {quote}",2 DM-1526,"Update processFile to use new measurement framework","processFile.py uses the old measurement framework in meas_algorithms, but all the other components expect it to be using the new measurement framework in meas_base.",4 DM-1527,"Draft security risks into the Center's template","The Cyber security center has provided a risk template consistent with their templates, The Center attempted to populate their templates with material from LSE 99 to the templates, The impedance mis match was too large, The way forward is seem as attempting a high level decomposition of risk into the templates.",4 DM-1528,"Identify potential KVM hardware","Identify potential KVM hardware that would meet our needs. e.g. a current version of the Avocent or Dell KVMs used at NCSA.",1 DM-1529,"Reorganize docker image repositories and align with github","A heterogeneous collection of docker images have been accumulating within the public docker repository daues/lsstdistrib . Such a heterogeneous collection prevents the assignment of a ""latest"" tag to allow users to easily obtain the most recent image for a particular item (detailed version numbers, labels currently required.) Thus we should break out the single repository into multiple repositories where are ""latest"" tag will be effective. We also make github repositories of matching names to hold the Dockerfiles which produced images (a common pattern for github/dockerhub usage, especially with automated builds; so we start this practice.) ",2 DM-1530,"Test if IPMI can replace KVM","Can we use IPMI in place of a KVM? Can we reliably do the following across our various server vendors? - power cycle hung systems - view and use text console - view and use remote gui console - boot from a remote ISO",3 DM-1531,"Create a LSST base CentOS image",NULL,2 DM-1532,"Document use of LSST image for CentOS",NULL,1 DM-1533,"Gather Open Stack needs/requirements from DM team",NULL,9 DM-1534,"Assist DM team in accessing test open stacks",NULL,1 DM-1538,"Fix qserv_testdata documentation","qserv_testdata relies on sconsUtils, and its build procedure has to be clearly documented.",1 DM-1539,"Add support for mysql JDBC driver","SUI which rely on JDBC fail because they internally issue some queries that are not yet supported by Qserv. Need to patch it (in the short term), and add proper support (in the long term). This story covers the patching only. The queries that upset Qserv are listed below. {code} SHOW VARIABLES WHERE Variable_name ='language' OR Variable_name = 'net_write_timeout' OR Variable_name = 'interactive_timeout' OR Variable_name = 'wait_timeout' OR Variable_name = 'character_set_client' OR Variable_name = 'character_set_connection' OR Variable_name = 'character_set' OR Variable_name = 'character_set_server' OR Variable_name = 'tx_isolation' OR Variable_name = 'transaction_isolation' OR Variable_name = 'character_set_results' OR Variable_name = 'timezone' OR Variable_name = 'time_zone' OR Variable_name = 'system_time_zone' OR Variable_name = 'lower_case_table_names' OR Variable_name = 'max_allowed_packet' OR Variable_name = 'net_buffer_length' OR Variable_name = 'sql_mode' OR Variable_name = 'query_cache_type' OR Variable_name = 'query_cache_size' OR Variable_name = 'license' OR Variable_name = 'init_connect' SELECT @@session.auto_increment_increment SET NAMES latin1 SET character_set_results = NULL SET autocommit=1 SET sql_mode='STRICT_TRANS_TABLES' {code}",2 DM-1540,"Add support for BIT columns","The mysql/SchemFactory code has incomplete logic for dealing with columns that have BIT type - in particular, it doesn't properly handle the length of a bit field, and it sets the SQL type to ""BIT?"", which causes table creation failures later on. Fixing this requires modifying SchemaFactory, and making sure that bit values are transmitted properly.",4 DM-1541,"Add support for transmitting [VAR]BINARY column data","The code that pulls data out of a {{MYSQL_ROW}} and puts it into a protobuf {{RowBundle}} does not handle binary data correctly. See [_fillRows|https://github.com/LSST/qserv/blob/master/core/modules/wdb/QueryAction.cc#L217] for the relevant code. The issue is that the generated {{add_column(const char*)}} member function of {{RowBundle}} treats the input as C-style NULL-terminated strings. But in the case of {{BINARY}} column data (and {{VARBINARY}}/{{BLOB}} variants, maybe also {{BIT\(n\)}}), the contents can contain embedded NULLs. We are currently using such columns for user defined types in MySQL (e.g. image bounding polygons), so it's important to get this right. On the protobuf side the fix is as simple as calling {{add_column(const char* value, int size)}} instead. I'm not a MySQL C API expert, but the size will presumably have to be obtained/derived from the corresponding {{MYSQL_FIELD}}. ",8 DM-1542,"Drawing speed will detect and optimize for large datasets",NULL,10 DM-1543,"Finished Background conversion",NULL,2 DM-1544,"Set up GIT hub for ipac firefly",NULL,18 DM-1545,"Span-based shrink operations for Footprint","Analogous to DM-1128, but shrinking rather than growing footprints.",5 DM-1547,"Install docker in an ISL OpenStack Ubuntu instance, perform basic checks",NULL,1 DM-1551,"Prototype HTM-based spatial binning to visualize large number of catalog sources","Currently we are using a generic 2-d binning algorithm, that is finding minimum and maximum values of the two columns to be visualized and bins the values into 2-d grid with the specified number of grid cells. This algorithm distorts data in the pole regions and whenever data are on both sides of ra=0. More smart binning based on a spatial index is necessary when reducing the number of (ra,dec) entries intended for visualization. ",16 DM-1552,"Resolving QServ database configuration/connectivity issues","To start the development I should be able to connect to the QSERV development database via VPN and run simple queries. Ideally, I'd like to be able to connect to QSERV with JDBC, view the data, run spatial queries. I also need an access to data dictionary to interpret data.",10 DM-1553,"Evaluate SDSS catalog access","SDSS allows two ways to access their catalog data: via HTTP Post service, accessible to anonymous users, and via CasJobs services, which require having an account. Evaluate how SDSS catalog data can be accessed from an application by prototyping single and multiple target searches accessing SDSS services. ",10 DM-1556,"FY17 Add Support for User Upload Tables for Qserv","Users will need to be able to upload a list of ""things"" to workspace, then request ""repeat a given query for each ""thing"" from the uploaded list"". Example of ""things"": ra/dec points, ra/dec points + distances, object ids, object names, bounding boxes, etc. In IPAC terminology, this is called ""table upload queries"". Note, we will need to assign some sort of unique id to such list. Results must contain information which result correspond to which ""thing"". E.g, if user asks for neighbors near (1,1), (4,4), it is not enough to just return list of neighbors, we need to tell which neighbor is for which point. In Qserv, that means that user will upload a special table with these ""things"", and then issue a query that involves join against that table. We need to evaluate and agree on how generic these upload tables can be. We might need to do some special optimizations (e.g., related to spatial searches). ",53 DM-1557,"FY17 User Upload Tables for ImgServ","Users will need to be able to upload a list of ""things"" to workspace, then request ""repeat a given query for each ""thing"" from the uploaded list"". Example of ""things"": ra/dec points, ra/dec points + distances, object ids, object names, bounding boxes, etc. In IPAC terminology, this is called ""table upload queries"". Note, we will need to assign some sort of unique id to such list. Results must contain information which result correspond to which ""thing"". E.g, if user asks for neighbors near (1,1), (4,4), it is not enough to just return list of neighbors, we need to tell which neighbor is for which point. In Qserv, that means that user will upload a special table with these ""things"", and then issue a query that involves join against that table. We need to evaluate and agree on how generic these upload tables can be. We need to decide on format / location: csv? Table in database? We might need to do some special optimizations (e.g., related to spatial searches).",53 DM-1560,"A better coordinate grid overlay","Some of the grid lines are not drawn right and the labels are not in the locations.",10 DM-1561,"Research Javascript Frameworks: General Overview","Begin looking into JS frameworks. Start to look into AngularJS, try to write some sample code. Attempt to understand the main concepts. Gat an overview of the others out there. ",12 DM-1562,"Work with the database group to define initial concepts for backend interface API ","We are starting to gel around some ideas. ",10 DM-1563,"Improve management of integration test datasets description","Currently data set description (i.e. data file extension, compressed extension, schema file extension) is hard-coded in python/lsst/qserv/tests/datareader.py This could be improve (standard format for test dataset, meta service, meta-configuration file, ...)",5 DM-1564,"Implement drawing only active tab with new data model",NULL,6 DM-1566,"LOE - Week ending 11/07/14",NULL,7 DM-1567,"LOE - Week ending 11/14/14",NULL,6 DM-1568,"LOE - Week ending 11/21/14",NULL,8 DM-1569,"LOE - Week ending 11/28/14",NULL,12 DM-1570,"Create integration test case using data duplicator","Integration tests should provide a new test case which use sph-duplicate in partition package.",6 DM-1571,"Setup Qserv for SUI tests","Setup Qserv on lsst-db2 with and load some reasonable data set (perhaps PT 1.2). One potential caveat: we need to setup access for some accounts that are ideally other than our internal qsmaster.",2 DM-1572,"Test deblended CModel colors","Using HSC data, examine color-color and color-magnitude diagrams with deblended CModel magnitudes. Investigate outliers by looking at images and deblended HeavyFootprints.",8 DM-1573,"Basic validation of LSST pipeline on HSC data","Get the pipeline running on HSC data to the point where nothing is obviously wrong.",8 DM-1574,"add support for ""freeze-drying"" measurement failures","We should make it easy for users to inspect and capture problems in measurement algorithms into an on-disk format that allows them to be reproduced later with minimal setup (ideally, the package would require no access to the original data or configuration). While this issue is mostly concerned with capturing problems in measurement algorithms, an extension to capture deblender failures should be considered as a future extension.",6 DM-1575,"Support Mac OS in scisql-deploy.sh","cf. Andy Connolly message: {quote} Just as an FYI on my mac scisql-deploy.py was looking for 0.3.4/lib/libscisql-scisql_0.3.so but there is only 0.3.4/lib/libscisql-scisql_0.3.dylib {quote}",1 DM-1576,"Sanitize AstrometryTask interface","Currently the AstrometryTask and Astrometry class share work. E.g. distortion is done in AstrometryTask but matching is done in Astrometry. AstrometryTask also makes assumptions about what fields are available in the solver config. The AstrometryTask interface should be sanitized so that it can be used as a thin wrapper for calling any astrometry solver. Top level config params should go in the AstrometryTaskConfig and solver level work should be done in the solver class and configured at the solver level.",10 DM-1577,"Rework Astrometry class","The Astrometry class shares information upstream with AstrometryTask. This should be factored out so that the Astrometry class can be configured and called via a single well known method (solve?). One thing the Astrometry class that is needed by down stream processing (PhotoCal) is to match sources. This is currently a private method, but should be made public so that it can be used without running AstrometryTask.",4 DM-1578,"Move photocal out of meas_astrom","It is confusing that photocal is in meas_astrom. I assume that is historical. I think it could probably live in pipe_tasks.",2 DM-1579,"Implement replacement for A.net index files","Astrometry.net index files are hard to generate and hard to read. We need another, more flexible, more standard system for storing reference files. We should also be able to read FITS files and other formats, but having a standard format with the utilities to create and query them is a must. Coming up with a format that satisfies astrometry.net's solver may be too hard, because a.net requires quads, which a non-blind solver may not need. However, a format that we can convert to something suitable for a.net would probably suffice (conversion would presumably be a separate task that is run once). It will be easier to identify a suitable format once we have identified at least one solver other than than a.net that we wish to implement or adapt. hscAstrom appears to use a catalog of star positions, which is nice and simple.",5 DM-1580,"Implement a replacement solver to the current A.net solver","This should be further refined. The solver will be required to work with several input formats. It will only be required to solve in the in the presence of a reasonable starting point with reasonable pointing errors. Failure should be graceful. If multiple, equivalent solutions are found, this should be reported (for situations including perfect grids of sources).",15 DM-1582,"Qserv spatial restrictor names are case sensitive","The SQL grammar treats Qserv spatial restrictor names case insensitively, but {{qana/QservRestrictorPlugin.cc}} does not, which means that one must use e.g. {{qserv_areaspec_box}} rather than {{qserv_areaSpec_box}}. We are loose with case in a lot of our wiki pages, so we really should fix this to avoid confusing users. Also, case insensitivity is consistent with MySQL behavior for UDF names.",1 DM-1584,"Research how to integrate different image metadata stores with DataCat","Data Release Production will generate highly structured image metadata (exposure* tables). If we decide to use DataCat (e.g., for keeping more ad hoc metadata), the question arises if/how to integrate all this together: * should we integrate all exposure* tables from all releases into DataCat? (eg via foreign tables) * should we keep them distinct, and integrate at higher level (e.g., Metadata Service)",4 DM-1585,"Design system for tracking existing images/files","We have a lot of files/images already brought in or generated through Data Challenges done to date. We need a system for cataloging them. This story will define such system, eg, sketch of UI, underlying technology used (DataCat, plain mysql, schema etc).",6 DM-1587,"Define structure of web form for collecting metadata about existing data sets","Web alpha version of the form (using django or Fermi Java webservices code) that collects input from users about data repositories. Authentication not covered in this version.",2 DM-1588,"Implement FITS header crawler and integrate it with the form","Implement crawler that walks through registered repos (through the form) and loads metadata from fits headers into the mysql backend",7 DM-1589,"Research and experiment with building form for capturing user input","Need to build a form that will be used to capture user input about existing image repositories (users will be registering their files/repositories). Options to consider: python-based django, java-based system that is part of Fermi DataCat. ",4 DM-1590,"Break down & discuss DM-1074","I will take the lead on DM-1074. First step will be to sit with [~jbosch], get a feeling for what needs to be done, and sketch out a set of stories.",1 DM-1591,"Convert test_qservAdmin.py into a real unit test","Need to turn ./admin/tests/test_qservAdmin.py into a real unit test. In the past it was broken and it went unnoticed, see DM-1395",1 DM-1594,"Remove check for stack dir write access in qserv-configure.py","qserv-configure.py checks for write access to stack dir, this should be replace by check for read access, or removed. fjammes@qserv-build-server-xlarge:~/src/qserv$ qserv-configure.py --all INFO: Qserv configuration tool ======================================================================= WARNING : Do you want to erase all configuration data in /home/fjammes/qserv-run/2014_10.0 ? [y/n] y INFO: Copying template configuration from /home/fjammes/stack/Linux64/qserv/2014_10.0/cfg/templates to /home/fjammes/qserv-run/2014_10.0 INFO: Creating meta-configuration file: /home/fjammes/qserv-run/2014_10.0/qserv-meta.conf INFO: Reading meta-configuration file /home/fjammes/qserv-run/2014_10.0/qserv-meta.conf INFO: Defining main directory structure INFO: No write access to dir /home/fjammes/stack/Linux64/qserv/2014_10.0 : [Errno 13] Permission denied: '/home/fjammes/stack/Linux64/qserv/2014_10.0/write_tester' ",2 DM-1595,"Research Javascript Frameworks: Understand Angular & React","Write some prototype code with Angular and then try to do the same thing with React",10 DM-1596,"Clean up multi-component Footprints","Following the landing of DM-1545, it's possible for an erosion operation on a footprint to cause it to split into multiple components and for peaks which were previously inside the footprint to not fall inside any of those components. Here, we should provide a ""clean up"" operation that takes a multiple-component footprint and splits it into a set of contiguous footprints with appropriate peak lists.",4 DM-1597,"init script fails to start xrootd after crash","I think we saw this issue in the past, not sure it was actually fixed back then or just was to reintroduced one more time. After crash of xrootd the regular {{etc/init.d/xrootd start}} fails to start it: {code} [salnikov@lsst-dbdev2 dm-621]$ ~/qserv-run/dm-621/etc/init.d/xrootd status xrootd is dead but PID file exists [FAILED] see /usr/local/home/salnikov/qserv-run/dm-621/var/run/worker/xrootd.pid [salnikov@lsst-dbdev2 dm-621]$ ~/qserv-run/dm-621/etc/init.d/xrootd start Starting xrootd: (already up) [ OK ] [salnikov@lsst-dbdev2 dm-621]$ ~/qserv-run/dm-621/etc/init.d/xrootd status xrootd is dead but PID file exists [FAILED] see /usr/local/home/salnikov/qserv-run/dm-621/var/run/worker/xrootd.pid {code} I can start it with {{restart}} but I think that {{start}} should detect that xrootd is dead ({{status}} does that) and start service correctly. This issue may exist for other services but I did not check. ",1 DM-1598,"Design of calibration and ingest system","Produce a confluence page describing the approach to be taken to solve DM-1074. Ensure that all the relevant folks have reviewed that page and are happy. Break down DM-1074 into stories appropriate to that design.",10 DM-1600,"Determine if Astrometry class is desired","The question is whether the Astrometry class is the thing that is overridden or if the AstrometryTask has component configurables that are overridden. Also, determine location default implementation.",2 DM-1601,"Add support for c-style comments in front of queries sent to qserv","Connecting to qserv from java fails because the jdbc driver inserts comments. ""/* ... */"" in front of queries (example pasted below). The fix involves removing the comments at the proxy level {code} SQLException: Qserv error: 'ParseException:ANTLR parse error:unexpected token: /:' Query being executed when exception was thrown: /* mysql-connector-java-5.1.34 ( Revision: jess.balint@oracle.com-20141014163213-wqbwpf1ok2kvo1om ) */SHOW VARIABLES WHERE Variable_name ='language' OR Variable_name = 'net_write_timeout' OR Variable_name = 'interactive_timeout' OR Variable_name = 'wait_timeout' OR Variable_name = 'character_set_client' OR Variable_name = 'character_set_connection' OR Variable_name = 'character_set' OR Variable_name = 'character_set_server' OR Variable_name = 'tx_isolation' OR Variable_name = 'transaction_isolation' OR Variable_name = 'character_set_results' OR Variable_name = 'timezone' OR Variable_name = 'time_zone' OR Variable_name = 'system_time_zone' OR Variable_name = 'lower_case_table_names' OR Variable_name = 'max_allowed_packet' OR Variable_name = 'net_buffer_length' OR Variable_name = 'sql_mode' OR Variable_name = 'query_cache_type' OR Variable_name = 'query_cache_size' OR Variable_name = 'license' OR Variable_name = 'init_connect' {code}",1 DM-1602,"Prevent collisions in /tmp related to scisql deployment","Deploying scisql involves creating a file in /tmp. The file never gets removed. This can cause the following error when Qserv is installed later on the same machine: {code} ERROR: failed to open output file /tmp/scisql_demo_htmid10.tsv for writing {code} We should switch to using a temporary file instead of file with fixed name.",2 DM-1603,"Qserv scons scripts do not pick up the version of swig provided by eups","See the summary. The consequence is that the Qserv integration tests fail on systems that provide swig 2.x+ - there seems to be some implicit dependency on SWIG 1.x. The reason may be that swig is getting confused about shared_ptr to objects that are defined in one module, but used in another (recent swig reorganization split czar and css into two separate swig modules).",2 DM-1604,"Sanitize configs","Some solver specific information is stored in the AstrometryTask config. Further, the solver config is accessed inside the AstrometryTask run method. This mixing of information make it hard to make the framework pluggable. Solver configuration should be confined completely within the solver class (whether it be part of the Astrometry class or a configurable of its own).",2 DM-1607,"Add unit tests to test c-style comments in/around queries","I should have thought about it when doing DM-1601 but I didn't... it'd be good to add test queries that test comments before/after/inside query.",1 DM-1608,"Move meas_algorithms unit tests to meas_base framework","The following tests in meas_algorithms need to be ported to the meas_base framework: measure.py psfSelectTest.py testPsfDetermination.py ticket2019.py testCorrectFluxes.py (though this cannot be done until the algorithm exists)",2 DM-1609,"Research off-the-shelf solutions for harvesting metadata","Relevant links: * http://www.ivoa.net/documents/SIA/20141024/index.html - there’s a list of metadata in the document * CAOM (Common Archive Object Model): http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2/",4 DM-1610,"Integrate Metadata Store prototype v1 with cat and db ","Integrate prototype developed through DM-1255 with cat and db as appropriate (don't hardcode schema, don't hardcode credentials, etc).",4 DM-1611,"Experiment with DataCat foreign tables","This is a placeholder story, we should break it down into more smaller stories...",15 DM-1614,"Add support for mysql JDBC driver (v2)","MySQL client 4.1 and higher is stripping out comments before sending them to server, so the fixes done in DM-1539 are not sufficient.",1 DM-1615,"Design and implement CSS structure for distributed Qserv setup","For management of the distributed databases/tables we need info in CSS about all workers and tables. The info will be created by data loader and updated by replicator which do not exist yet. For this issue we need to provide python API which can fill the same information in CSS so that we can build and test other pieces needed for this epic.",5 DM-1616,"Implement remote host access for management framework","To manage remote workers we need a way to access services on remote machines that run workers. Services may be something like mysql (which we would prefer to run without TCP port open) and optionally xrootd. This ticket will implement Python module which will hide a complexity of doing things like ssh/port forwarding/authentication from the client.",5 DM-1617,"Client library for accessing distributed database/table information from CSS","Provide Python interface for accessing information in CSS which is relevant to distributed management, such as database/table/node data. This interface can be used also by data loader and replicator.",8 DM-1618,"Implement distributed database creation","Implement Python library which creates databases on all active workers based on info from CSS.",10 DM-1619,"Implement distributed table creation","Implement Python library for creating mysql tables on all active workers.",11 DM-1620,"Move CSS documentation close to code ","CSS documentation about the structure is currently in trac (which is readonly), at https://dev.lsstcorp.org/trac/wiki/db/Qserv/CSS. We need to move it close to the source code, e.g., to a doc file.",2 DM-1621,"Add unit test to to verify zookeeper versioning works","This is a follow u pto DM-1453, we need to add a unit test that will prove that mismatched versions in zookeeper and software are properly handled.",1 DM-1622,"Add unit test for queries from DM-1539","Need to add unit test for queries listed in DM-1539",1 DM-1624,"Qserv should report it's version","It should be possible to determine which version of qserv we are running by looking at information from log files. So, in practice, we probably should generate in scons a unique id (from sha from git?) and compile it into the code.",1 DM-1625,"SciSQL should report its version","It should be possible to determine which version of scisql we are running.",1 DM-1626,"Build 2014_12 Qserv release","The title says it all. Please open ticket for next release when closing this one.",1 DM-1627,"Integrate metaserv, imgserv and dbserv with Data Access Services","* Create dbserv for handling database related web-based queries, support running queries through web (initial version). * Integrate dbserv, metaserv and imgserv RESTful API into the webserv - want to run all services (meta, image, db) under one server. * Also want to be able to run them separately, so have some handy servers for each service * Proof of concept for supporting multiple formats (html, json)",22 DM-1629,"Adopted/Retired RFCs are not counted as resolved","Also, marking an RFC as Adopted brings up a box with a message applicable to the Closed status.",1 DM-1630,"New RFCs should result in dm-devel E-mails and HipChat postings","Email notices of new RFCs filed with a DM component go to dm-devel Email notices of new RFCs filed with a DM component go to Data Management chat room Email notices of all new RFCs go to Bot: RFC room ",1 DM-1632,"Build 2014_11 Qserv release","The title says it all. Please open ticket for next release when closing this one. Tasks to do: - publish last buildbot build under a temporary eups-tag (""qserv-dev"")and test it, if it works fine: - create git-tags for Qserv and dependencies - publish the release with eups-tags ""qserv"" and ""YYYY_MM"" - generate and publish the doc for release ""YYYY_MM"" - update release number in Qserv code and set ""YYYY_MM+1"" as release in dev and ""YYYY_MM"" as stable release (update {{admin/bin/qserv-version.sh}}, {{doc/source/conf.py}}, {{doc/source/toplevel.rst}}) - generate and publish the doc for release ""YYYY_MM+1"" - look at the doc - commit {{admin/bin/qserv-version.sh}}, {{doc/source/conf.py}}, {{doc/source/toplevel.rst}} with current ticket number This procedure should be validated and documented.",1 DM-1633,"Update build process for Firefly opensource","Update build and deploy related scripts to reflects the changes affected by open sourcing of Firefly.",7 DM-1635,"Remove redundant CORS headers from Firefly's http response","Make sure CORS related headers are not sent when the Origin header is missing. Firefox does not like it.",1 DM-1636,"Research popular web development technologies","Research popular web development technologies to prepare Firefly for the future with the focus on front-end framework. ",6 DM-1641,"Document ""getting started"" procedure for new stack developers","Document a procedure for building the stack on a new system in a way that is appropriate for both project members and external contributors. This can be linked as a ""getting started"" guide from http://dm.lsst.org/.",5 DM-1642,"LOE - Week ending 12/5/14",NULL,8 DM-1645,"preparation work for FIrefly open source","1. discuss with IPAC director and managers to open source Firefly 2. study the major open source license,Apache, GPL, BSD 3-clause, MIT. 3. final decision: BSD-3 clause",6 DM-1647,"Use an OpenStack instance to run an HTCondor Central manager","Use an OpenStack instance to run an HTCondor Central manager",4 DM-1648,"W16 Research technologies for Data Access","Research technologies potentially useful for Data Access / Database. This epic covers Apache Mesos, Google Kubernetes, MaxScale, Serf, Consul, and MemSQL.",38 DM-1652,"The existing FITS reader class needs to be refactored to improve the performance(1)","- checkout the classes - understand the FITS standard and current implementation",16 DM-1653,"Extend data loading script to support multi-node setup","Implement simplest use case for data loading with one or more worker database separate from czar database. Simplest means minimal functionality in what concerns access to workers, just assume for now that we can connect to every worker directly using regular TCP connection. It should be possible to just add a list of worker nodes as an option to loader script and send the chunks in some random order to the workers in that list. Of course chunks for different tables should end up on the same host, so some form of chunk management is needed (use for now whatever is defined in CSS doc on trac).",8 DM-1655,"Working with SLAC on definition of metadata store","Follow up metadata store schema development to ensure SUI will be able to use it. Define the fields that should go into Data Definition table. Define the fields that must be present in the image metadata table, which SUI will be searching.",2 DM-1656,"hipchat support for maigically urlifying docushare documents","It would be generally use full if references to official documents in hipchat (or it's successor) automagically generated urls for official document handles. Eg: Document-1234, LSE-123",1 DM-1657,"LOE - Week ending 12/12/14",NULL,8 DM-1658,"Add git bisect tool for Qserv repos","{code:bash} fjammes@clrlsst-dbmaster-vm:~/src/qserv (u/fjammes/DM-627 $%) $ qserv-test-head.sh -h Usage: qserv-test-head.sh [options] Available options: -h this message -q quick: only rebuild/install new Qserv code, and perform test case #01 Rebuild from scratch, configure and run integration tests against a Qserv git repository. Pre-requisite: source loadLSST.bash setup qserv_distrib -t qserv setup -k -r ${QSERV_SRC_DIR} Can be used with 'git bisect' : cd ${QSERV_SRC_DIR} git bisect start git bisect bad git bisect good git-commit-id git bisect run /home/fjammes/src/qserv_testdata/bin/qserv-test-head.sh {code} Code is in DM-627 ticket branch.",2 DM-1659,"Aperture Flux back into an abstract class","The last change to ApertureFlux to make it work with the new C++ design changed ApertureFluxAlgorithm into an instantiatable class. However, I have now figured out how to make this work with SWIG while still allowing measure and fail to be defined by default at the ApertureFlux level.. So this issue is to put things back in order.",1 DM-1660,"Statistics tests use a constant image","I just noticed that the tests for Statistics (clipped mean etc.) use a constant image. We should be testing a Gaussian field (although that makes the tests a little trickier) ",2 DM-1661,"czar log4cxx link/load bug","Under ubuntu 14.04 (at least), the czar falls over at load time with an unresolved sym for typeinfo for log4cxx::helpers::ObjectPtrBase while loading the css python wrapper shared lib.",2 DM-1662,"Make qserv dependencies build on OS X with clang","Fix anything necessary for qserv dependencies to build on OS X with clang. Note -- making qserv itself build is more complicated and may require a separate ticket.",4 DM-1663,"fix dependency problems in obs_subaru scons scripts","When building obs_subaru with -j4, it often tries to build files related to the defects before the main Python module is built, resulting in import errors (because the scripts it invokes depend on the main Python module). We need to rewrite the SCons scripts to ensure this dependency is captured. A preliminary look indicated that this is not entirely trivial, and I'll have to remind myself a bit of how some things in SCons work to get it done, so I'm putting this off for a future sprint. In the meantime, the workaround is to build obs_subaru with no parallelization.",2 DM-1664,"Understand historical written docushare materials dealing with the operational security environment.","researched the docushare traversing the plans for materials to be embedded in the OCS and similar systems, as well as extant operational plans. The goal was to understand how to separate the the security responsibilities of development, what security constraints ought to be but on items that are delivered, and what to tell the camera and telecscope teams to mender ea smooth integration with IT security systems upon delivey and integration of their sub systems in Chile.",3 DM-1667,"Install PgMySQL and use to connect to local Qserv.","Used ""pip"" to install it. ""Conda"" should work as well. Therefore, it should be easy to make it part of the delivered system: VM, container, tar file, after the fact download, etc. It has documentation, uses the MIT license, under active development and available from PyPI. DB connection is straight forward and requires little experience to get meaningful work done.",2 DM-1668,"Create SQL code to read Qserv into Python Pandas data frame","This works well, at least for a simple case. You can move directly from a query statement to a Pandas data frame for analysis in just a few lines of code. Here is the start of an iPython Qserv session showing how easy it is. In [6]: import pandas as pd In [7]: import pymysql as db In [8]: conn = db.connect(host='lsst-db1.ipac.caltech.edu',port=4040, user='qsmaster', passwd='', db='LSST') In [11]: df = pd.read_sql(""select deepCoaddId, tract, patch, ra, decl from DeepCoadd"", conn) In [12]: df Out[12]: deepCoaddId tract patch ra decl 0 26607706 0 406,11 0.669945 1.152218 1 26673242 0 407,11 0.449945 1.152218 2 26804242 0 409,2 0.011595 -0.734160 3 26673154 0 407,0 0.449945 -1.152108 … ",2 DM-1669,"Explore queries for Qserv database.","Use a local database here at IPAC with 5 Qserv tables in it. Looked at several Python query interfaces. Used the pymysql interface for testing because it's pure Python and because I found reports suggesting it was almost as fast as the mysqldb interface that requires C language libraries. Ad hoc queries can be constructed in three lines of code, so useable in a science environment. Found a couple of bugs in Qserv that were reported.",6 DM-1670,"Begin looking at how Python Pandas can be used for LSST data analysis.","Pandas is well integrated with the other parts of SciPy: numpy, matlibpy, etc. It’s a good candidate for data analysis, especially where time series are involved. However, there are no multidimensional columns, poor metadata support for FITS files and a need to use masks instead of NaN values. These may, or may not, be problems. There is a 400 page book about Pandas, so it will take some further time to learn its value, especially with astronomical data in different situations.",5 DM-1673,"Allow SWIG override for broken SWIG installations","Dependency on SWIG 2.0+ was introduced into Qserv, and this broke Qserv building on systems relying on SWIG 1.3.x. This ticket introduces basic code to override SWIG_LIB on those systems to allow use of the broken installation (some SWIG search paths are fixed during its build process otherwise).",1 DM-1674,"Data ingest scripts cleanup",NULL,10 DM-1675,"Identify dead code",NULL,1 DM-1676,"Write documentation for SSI interface",NULL,5 DM-1677,"multi-error utility class","QueryAction::Impl currently has a handful of private members/methods related to maintaining a collection of errors that occurs while processing a query (_addErrorMsg, IntString, IntStringVectory, _errors, etc.) It would be useful to split this stuff out into a separate, re-usable utility class, and extend, then extend with some additional functionality (output stream operator, subclass from std::exception, maybe capture __LINE__ and __FILE__, integrate with logging, etc.) [Fabrice: discuss requirements and basic design w/ Fritz and/or Daniel]",10 DM-1685,"Minor bug in a test","tests/centroid.py has a bug in testMeasureCentroid: ""c"" is undefined in the following bit of code: {code} if display: ds9.dot(""x"", c.getX(), c.getY(), ctype=ds9.GREEN) {code}",1 DM-1692,"Implement ""unlimited"" result size handling","DM-854 exposed an issue in handling large results. Result rows are returned from worker to czar in protobufs messages. However, protobufs messages should not be larger than some number of megabytes, according to protobufs documentation. IOW, protobufs is not designed to handle messages on the order of hundreds of megabytes. There may be some code in the protobufs implementation that does not scale beyond messages of a few megabytes. Hence, in order to send larger amounts of result rows, we need to use multiple messages. The current protobufs definition for result messages includes a placeholder for chaining result messages, but there is no code on the czar or worker that implements result chaining. The scope of this ticket is to implement message chaining for results on the worker and on the czar, in such a way that it places no limits on the overall number of result rows (if there is a limit, it should be no smaller than quadrillions and be well-documented). ",15 DM-1694,"Define interfaces for Data Access Services",NULL,8 DM-1695,"Implement interfaces for Data Access Services","Implement proof of concept, skeleton of the prototype. The work will continue in follow up stories in February and in S15.",8 DM-1696,"Integrate image cutout service interfaces with butler",NULL,5 DM-1697,"Finish image cutout service implementation","Define appropriate interfaces and connect them with the RESTful API (see DM-1695).",9 DM-1698,"Finish metadata store prototype","* A quick proof-of-concept prototype of loading tool for loading database-information into metadata store. * Handling mysql credentials through auth file in home dir instead of hardcoded values.",4 DM-1700,"Create read only OpenStack volume and execute processing scenario","Create read only OpenStack volume and execute processing scenario",4 DM-1701,"Clone OpenStack volume for use against multiple instances","Clone OpenStack volume for use against multiple instances",2 DM-1702,"Examine fqdn/hostname assignment for OpenStack instance","Examine fqdn/hostname assignment for OpenStack instance",4 DM-1703,"S15 Implement Database & Table Mgmt","Continuation of DM-1036, making the code for managing distributed databases and tables more feature reach, including features such as deletion.",53 DM-1704,"S15 Run Large Scale Qserv Tests","Run large scale tests to uncover unexpected issues and bottlenecks.",48 DM-1705,"S15 Tune Qserv","Fix scalability and performance issues uncovered through large scale tests DM-1704",100 DM-1706,"S15 Analyze Qserv Performance","Final analysis of Qserv performance, measure KPIs. Based on LDM-240, we are aiming to demonstrate: * 50 simultaneous low volume queries, 18 sec/query * 5 simultaneous high-volume queries, 24 h/query * data size: 10% of DR1 level. * Continuous running for 24 h with no software failures. ",5 DM-1707,"S15 Refactor Qserv","Ongoing refactoring of Qserv - code cleanup, tightening interfaces etc.",100 DM-1708,"W16 Improve Query Coverage in Qserv","Currently Qserv supports only a limited subset of queries. We need to make sure it supports all queries that users need to run.",46 DM-1709,"Implement result sorting for integration tests","We need to be able to sort results, because we can't always rely on ORDER BY. So we need a formatting per query in the integration tests (sort result for some, don't sort for others etc.) The following queries have been disabled because we don't have result sorting, so once it is implemented, we will need to re-enabled them prior to closing this ticket: {code} case02/queries/0003_selectMetadataForOneGalaxy_withUSING.sql case02/queries/3001_query_035.sql case02/queries/3008_selectObjectWithColorMagnitudeGreaterThan.sql case02/queries/3011_selectObjectWithMagnitudes.sql case02/queries/3011_selectObjectWithMagnitudes_noalias.sql {code}",8 DM-1710,"ValueError in lsst.afw.table.Catalog.extend()","{code} from lsst.afw.table import BaseCatalog, Schema s = Schema() c1 = BaseCatalog(s) c2 = BaseCatalog(s) c1.extend(c2) {code} The above fails, saying: {code} Traceback (most recent call last): File ""test.py"", line 7, in c1.extend(c2) File ""/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/afw/10.0+3/python/lsst/afw/table/tableLib.py"", line 6909, in extend _tableLib.BaseCatalog_extend(self, iterable, deep) ValueError: invalid null reference in method 'BaseCatalog_extend', argument 3 of type 'lsst::afw::table::SchemaMapper const &' {code}",1 DM-1711,"S15 Improve MetaServ: RESTful, Basic Image Search, DDL, Config Files","Implement beta version of the Metadata Service. This version will support basic image search, DDL, config files, and RESTful interfaces for MetaServ and Database (Qserv).",57 DM-1712,"S15 Add Support for Image Stitching and Rotation, RESTful APIs","Implement image stitching and rotating, including RESTful APIs.",35 DM-1713,"S15 Image & File Archive v2","System for tracking existing image data sets integrated with metadata services.",5 DM-1714,"Integrate MetaServ with Schema Browser","Schema browser displays detailed info about schema, including custom fields like UCD, units etc. This information is stored as comments embedded in the master version of the schema (in ""cat"" repo). Currently we are generating ascii from the master schema for schema browser, and we load it into mysql, then schema browser reads it from mysql. This story involves changing schema browser such that it will read the schema information directly from MetaServ.",6 DM-1715,"Disable query killing","Apparently killing a query through Ctrl-C is confusing xrootd. Disable query killing (which seems to be only partly implemented).",1 DM-1716,"Implement query killing through Ctrl-C","Need to properly implement query killing through Ctrl-C",16 DM-1720,"Make secondary index for director table only","Following discussion on qserv-l, we only need to generate ""secondary"" index for director table, no other table is supposed to have it. Need to modify data loader to recognize which table is director table and generate index only for that table. ",2 DM-1721,"S15 Improve Query Coverage in Qserv","Query coverage in the qserv integration testing is very limited, we have been turning off more and more queries and we were making the qserv code and the data loader more strict. This epic covers work (fixes and improvements) related to * re-enabling test queries marked as ""fixme"" (when it make sense, some queries are for features that are not implemented yet) * adding more queries to test interfaces and features that are implemented but are not currently tested.",40 DM-1722,"LOE - Week ending 12/19/14","The System Administration team at NCSA worked on the following LOE tasks this week: - RMA'ed RAID card for lsst-dev - Updated user's updated public ssh keys - Rebuilt failed drive in lsst-dbdev2 - Increasing drive size of lsst-eval - Researched & repaired lsst20 NFS issues - Upgraded lsst-xfer to 10G networking",14 DM-1725,"stack build fails on gcc 4.8 with opt=3","The stack fails to build with gcc 4.8, with a test failure in meas_base (though a similar problem on the HSC fork suggests the problem is actually in afw). [~darko] reports that on OpenSuse 13.1, the failure goes away when compiling with opt=1 instead of the default opt=3, indicating that the problem is overly aggressive optimization. This, and the fact that a traceback of the HSC-side failure implicates MaskedImage::getXY0, leads me to guess that something is going wrong with the alignment of the Eigen data members in afw::geom::Point. Until that theory is disproven, this probably belongs in my court, though I'd be happy to let someone steal it from me if they're interested in working on it.",4 DM-1726,"Stabilize Firefly Repository","- Make GitHub firefly ready but not public - Add firefly to LSST git - Clean up any lingering issues from separation - break up IRSA Viewer from Firefly Viewer",14 DM-1727,"Research Javascript Frameworks: Work toward future proposal","Take prior research and come up with rough firefly migration proposal. Include how to use a hybrid system for foreseeable future. Write some prototype code.",16 DM-1731,"fix table file handling of MANPATH in dependencies","As discussed on DM-1220, the table files for: - mysqlproxy - protobuf - lua - expat should have the MANPATH entry removed entirely, while: - xrootd should have "":"" added to the end of its MANPATH value, to allow the default paths to be searched as well.",1 DM-1732,"Fix error on duplicate result_id_m table while launching qserv integration tests","Next command: {code} qserv-check-integration.py --case=01 --load; qserv-check-integration.py --case=02 --load {code} Fails, most of the time, with next error in logs: {code} qserv@clrinfoport09:~/src/qserv (u/fjammes/DM-627 *+)⟫ cat ~/qserv-run/2014_12/var/log/qserv-czar.log | grep result_1211906_m 0103 16:38:18.576 [0x7f23c4ff9700] DEBUG root (build/rproc/InfileMerger.cc:432) - InfileMerger create table:CREATE TABLE qservResult.result_1211906_m (`ra` DOUBLE) 0103 16:38:18.661 [0x7f23c4ff9700] DEBUG root (build/rproc/InfileMerger.cc:354) - InfileMerger sql success: CREATE TABLE qservResult.result_1211906_m (`ra` DOUBLE) 0103 16:38:18.661 [0x7f23c4ff9700] DEBUG root (build/rproc/InfileMerger.cc:445) - InfileMerger table qservResult.result_1211906_m is ready 0103 16:38:18.686 [0x7f23c77fe700] INFO root (build/rproc/InfileMerger.cc:299) - Merging w/CREATE TABLE qservResult.result_1211906 SELECT ra AS ra FROM qservResult.result_1211906_m ORDER BY ra 0103 16:38:18.720 [0x7f23c77fe700] DEBUG root (build/rproc/InfileMerger.cc:354) - InfileMerger sql success: CREATE TABLE qservResult.result_1211906 SELECT ra AS ra FROM qservResult.result_1211906_m ORDER BY ra 0103 16:38:18.720 [0x7f23c77fe700] INFO root (build/rproc/InfileMerger.cc:305) - Cleaning up qservResult.result_1211906_m 0103 16:38:18.721 [0x7f23c77fe700] INFO root (build/rproc/InfileMerger.cc:317) - Merged qservResult.result_1211906_m into qservResult.result_1211906 0103 16:38:24.716 [0x7fe3cf7fe700] DEBUG root (build/rproc/InfileMerger.cc:432) - InfileMerger create table:CREATE TABLE qservResult.result_1211906_m (`QS1_COUNT` BIGINT(21)) 0103 16:38:24.717 [0x7fe3cf7fe700] ERROR root (build/rproc/InfileMerger.cc:351) - InfileMerger sql error: Error applying sql. Error 1050: Table 'result_1211906_m' already exists Unable to execute query: CREATE TABLE qservResult.result_1211906_m (`QS1_COUNT` BIGINT(21)) 0103 16:38:24.717 [0x7fe3cf7fe700] ERROR root (build/rproc/InfileMerger.cc:438) - InfileMerger error: Error creating table (qservResult.result_1211906_m) 0103 16:38:34.672 [0x7fe3deffd700] INFO root (build/rproc/InfileMerger.cc:299) - Merging w/CREATE TABLE qservResult.result_1211906 SELECT SUM(QS1_COUNT) AS OBJ_COUNT FROM qservResult.result_1211906_m 0103 16:38:34.673 [0x7fe3deffd700] ERROR root (build/rproc/InfileMerger.cc:351) - InfileMerger sql error: Error applying sql. Error 1054: Unknown column 'QS1_COUNT' in 'field list' Unable to execute query: CREATE TABLE qservResult.result_1211906 SELECT SUM(QS1_COUNT) AS OBJ_COUNT FROM qservResult.result_1211906_m 0103 16:38:34.673 [0x7fe3deffd700] INFO root (build/rproc/InfileMerger.cc:305) - Cleaning up qservResult.result_1211906_m 0103 16:38:34.674 [0x7fe3deffd700] INFO root (build/rproc/InfileMerger.cc:317) - Merged qservResult.result_1211906_m into qservResult.result_1211906 {code} So it seems test case #01 create table .result_1211906_m, and then test case #02 try to re-use this name for an other query. Qserv result tables aren't cleaned (here Qserv has be stopped): {code} ls ~/qserv-run/2014_12/var/lib/mysql/qservResult/ db.opt result_1211336_m.MYI result_1211340_m.MYI result_1211343_m.MYI result_1211346_m.MYI result_1211382_m.MYI result_1211385_m.MYI result_1211902_m.MYI result_1211905_m.MYI result_1211334_m.frm result_1211337_m.frm result_1211341_m.frm result_1211344_m.frm result_1211347_m.frm result_1211383_m.frm result_1211386_m.frm result_1211903_m.frm result_1211906_m.frm result_1211334_m.MYD result_1211337_m.MYD result_1211341_m.MYD result_1211344_m.MYD result_1211347_m.MYD result_1211383_m.MYD result_1211386_m.MYD result_1211903_m.MYD result_1211906_m.MYD result_1211334_m.MYI result_1211337_m.MYI result_1211341_m.MYI result_1211344_m.MYI result_1211347_m.MYI result_1211383_m.MYI result_1211386_m.MYI result_1211903_m.MYI result_1211906_m.MYI result_1211335_m.frm result_1211339_m.frm result_1211342_m.frm result_1211345_m.frm result_1211381_m.frm result_1211384_m.frm result_1211901_m.frm result_1211904_m.frm result_1211335_m.MYD result_1211339_m.MYD result_1211342_m.MYD result_1211345_m.MYD result_1211381_m.MYD result_1211384_m.MYD result_1211901_m.MYD result_1211904_m.MYD result_1211335_m.MYI result_1211339_m.MYI result_1211342_m.MYI result_1211345_m.MYI result_1211381_m.MYI result_1211384_m.MYI result_1211901_m.MYI result_1211904_m.MYI result_1211336_m.frm result_1211340_m.frm result_1211343_m.frm result_1211346_m.frm result_1211382_m.frm result_1211385_m.frm result_1211902_m.frm result_1211905_m.frm result_1211336_m.MYD result_1211340_m.MYD result_1211343_m.MYD result_1211346_m.MYD result_1211382_m.MYD result_1211385_m.MYD result_1211902_m.MYD result_1211905_m.MYD {code} Changing _idCounter to next value in appInterface.py: {code:python} self._idCounter = int(time.time() % (60*60*24*365) * 10) {code} solves the problem, but a deeper explanation will allow to bring a more robust fix.",8 DM-1733,"Build 2015_01 Qserv release","See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe.",1 DM-1734,"refactor fftools viewer to have derived viewers / start LSST SUI git repo","Do the following: - refactor fftools to allow for specialized viewers. - make a generic fftools viewer. - make an irsaviewer package in ife-new that will build. - generalize the catalog search and add factories to access the lsst search process. ",10 DM-1735,"Have newinstall.sh check itself against distrib version","We want to alert people who are just using a newinstall.sh they have lying around (old or hacked up or...) that they are not using the official server version. ",1 DM-1736,"Migrate qserv modules (python code) to new logging system",NULL,8 DM-1738,"deblender artifacts in noise-replaced images","We still see noise artifacts in some deblended images on the LSST side when running the M31 HSC data. They look like the result of running NoiseReplacer on HeavyFootprints in which the children can extend beyond the parents. This was fixed on the HSC side on DM-340 (before the HSC JIRA split off), and I *think* we just need to transfer the fix to LSST.",1 DM-1739,"Catch-all epic for essential fixes during DM-W15-4",NULL,10 DM-1743,"CSV reader for Qserv partitioner doesn't handle no-escape and no-quote options properly","Both the no-quote and no-escape CSV formatting command line options should not have a default value, as specifying any value turns off field escaping and quoting. Furthermore, when quoting is turned off, the reader incorrectly treats embedded NUL characters as a quote character.",1 DM-1744,"Fix SWIG_SWIG_LIB empty list default value","See Serge message to Qserv-l ""xrootd premature death"": {quote} However, there are bigger problems. First of all, master doesn’t build for me. I get this error: File ""/home/lsstadm/qserv/SConstruct"", line 104: env.Alias(""dist-core"", get_install_targets()) File ""/home/lsstadm/qserv/SConstruct"", line 90: exports=['env', 'ARGUMENTS']) File ""/home/lsstadm/stack/Linux64/scons/2.3.0+1/lib/scons/SCons/Script/SConscript.py"", line 609: return method(*args, **kw) File ""/home/lsstadm/stack/Linux64/scons/2.3.0+1/lib/scons/SCons/Script/SConscript.py"", line 546: return _SConscript(self.fs, *files, **subst_kw) File ""/home/lsstadm/stack/Linux64/scons/2.3.0+1/lib/scons/SCons/Script/SConscript.py"", line 260: exec _file_ in call_stack[-1].globals File ""/home/lsstadm/qserv/build/SConscript"", line 39: canBuild = detect.checkMySql(env) and detect.setXrootd(env) and detect.checkXrootdLink(env) File ""/home/lsstadm/qserv/site_scons/detect.py"", line 225: xrdLibPath = findXrootdLibPath(""XrdCl"", env[""LIBPATH""]) File ""/home/lsstadm/qserv/site_scons/detect.py"", line 213: if os.access(os.path.join(path, fName), os.R_OK): File ""/home/lsstadm/stack/Linux64/anaconda/2.1.0/lib/python2.7/posixpath.py"", line 77: elif path == '' or path.endswith('/'): which is caused by the fact that env[“LIBPATH”] looks like: [[], '/home/lsstadm/stack/Linux64/antlr/2.7.7/lib', '/home/lsstadm/stack/Linux64/boost/1.55.0.1.lsst2/lib', '/home/lsstadm/stack/Linux64/log4cxx/0.10.0.lsst1+2/lib', '/home/lsstadm/stack/Linux64/xrootd/4.0.0rc4-qsClient2/lib', '/home/lsstadm/stack/Linux64/zookeeper/3.4.6/c-binding/lib', '/home/lsstadm/stack/Linux64/mysql/5.1.65.lsst1/lib', '/home/lsstadm/stack/Linux64/protobuf/2.4.1/lib', '/home/lsstadm/stack/Linux64/log/10.0+3/lib'] The first element is [], which comes from https://github.com/LSST/qserv/blob/master/site_scons/state.py#L173 where a PathVariable called SWIG_SWIG_LIB is given a default value of []. I can fix the build by changing the default to an empty string… but I don’t know enough scons to say whether that’s the right thing to do. Can one of the scons gurus confirm that’s the right fix? {quote}",1 DM-1753,"make lsst-sui & firefly git repo / make an lsst viewer",NULL,4 DM-1754,"Update auto build tool to work with new split repositories ","After the repository split, changes are required to get the auto build tool to work properly. Firefly and Firefly based applications are built using Gradle system. ",8 DM-1755,"Create an integration test case with GB-sized data","It's difficult to load manually data in Qserv, so a way to do that is to use integration test framework to automatically do this. Big data file won't be stored in git, but the user wil lhave to retrieve them manually, and the test case won't be executed by integration tests.",4 DM-1761,"Provide input data for exampleCmdLineTask.py","{{pipe_tasks/examples/exampleCmdLineTask.py}} reads data from a repository. The comments in {{pipe_tasks/python/lsst/pipe/tasks/exampleCmdLineTask.py}} suggest that {code} # The following will work on an NCSA lsst* computer: examples/exampleCmdLineTask.py /lsst8/krughoff/diffim_data/sparse_diffim_output_v7_2 --id visit=6866601 {code} There are a few problems with that: * External contributors don't have access to {{lsst*}}; * Even though that data exists now, it's unclear how long it will remain there, or what steps are being taken to preserve it; * The mention of this data is fairly well buried -- it does appear in the documentation, but it's certainly not the first thing a new user will stumble upon. At least the first two points could be addressed by referring to a publicly available data repository. For example, the following works once {{afwdata}} has been set up: {code} examples/exampleCmdLineTask.py ${AFWDATA_DIR}/ImSim --id visit=85408556 {code} Although this has the downside of only providing a single image.",1 DM-1762,"Export SUI data (DC_W13_Stripe82_subset)","- import sui.sql.bzip2.out (produced by Serge) into MySQL for DeepSource and DeepForcedSource tables: - remove columns chunkId and subChunkId for each chunk table - merge all chunk table into the main table - join DeepSource and DeepForcedSource to add coordinates of DeepSource (director) object in DeepForcedSource table. then dump DeepSource and DeepForcedSource to files DeepSource.csv and DeepForcedSource.csv {code:sql} SELECT f.*, COALESCE(s.ra, f.ra), COALESCE(s.decl, f.decl) FROM DeepForcedSource f LEFT JOIN DeepSource s ON (f.deepSourceId = s.deepSourceId) INTO OUTFILE '/db1/dump/DeepForcedSource.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '""' LINES TERMINATED BY '\n'; {code} - Load this file using Qserv loader. A sample should be made and tested first to validate this procedure. This sample could be added in qserv_testdata",3 DM-1764,"overhaul slot and alias system","While working on DM-1218 and DM-464, I've grown quite dissatisfied with the current state of the slot and alias mechanisms, and we now have a concrete proposal for larger-scale changes on RFC-11. Unfortunately, I don't think we'll be in a good position to do much about this until we've completed the transition to meas_base and removed the old measurement framework in meas_algorithms.",6 DM-1765,"move SourceRecord/Table/Catalog to meas_base","We can make address a lot of dependency issues if we move the Source classes to meas_base, because we'll no longer have low-level code (e.g. afw::table persistence) in the same module as very high-level code (e.g. slots). It will also put all the slot code in the same place, instead of spreading it across two pacakges. This should be straightforward, except that we'll have a lot of downstream code to (trivially) change, and there's a good chance Swig will get confused somewhere along the way.",2 DM-1766,"Remove in-memory support of old-version afw::table objects","After removing the old measurement framework in meas_algorithms, we should also end support for version=0 Schemas in memory, and instead convert version=0 Schemas to version 1 when we unpersist them.",11 DM-1767,"afw::table - post-transition improvements","Breakdown: jbosch 40%; swinbank 60%",23 DM-1769,"Measurement - Framework Improvements","Add new features in meas_base that are desirable, but not required to replace functionality in meas_algorithms. Breakdown: pgee 70%; jbosch 30%",60 DM-1770,"Support DDL in MetaServ - design","DDL information is embedded as comments in the master version of the schema (in ""cat"" repo). Currently we are only using it for schema browser. This story involves designing the procedure involving loading DDL information into MetaServ. We need to be ready to support a variety of scenarios: * we are getting already preloaded database, need to just load metadata about it to metaserv (we might have the original ascii file with extra information, or not) * we are starting from scratch, need to initialize database (including loading schema), and need to load the information to the metaserv * we already have the database and metadata in metaserv, but we want to change something (eg. alter table, or delete table, or delete database).",2 DM-1771,"move executionOrder from plugin config class to plugin class","We originally put the executionOrder parameter (which determines when a plugin is run, relative to others), in the config object, simply because that's where it was in the old framework. But it's really not something that should be configurable, as it depends only on the inputs the algorithm needs, which don't change.",1 DM-1772,"Create a search processor to do cone/box search on a QSERV catalog","Create a search processor which accepts cone and box spatial constraints, queries a catalog, stored on QSERV (DeepSource), and returns the relevant rows as an IPAC table or RawDataSet. The implementation is querying DeepSource catalog using qserv_areaspec_circle and qserv_areaspec_box spatial restrictors. ",11 DM-1773,"Read SUI requirements, send a list of questions to the group scientist","Read the requirements document on Archive Browser and Query Tools, make a record of unclear items, send questions to our scientist (D. Ciardi) ",5 DM-1774,"Add support for running unit tests in qserv/admin","This came up during review of DM-370: ""We do not run tests scripts in admin/ during regular build, SConscript in admin/ does not support that unfortunately."" This story involves tweaking SConscript to enable running unit tests automatically.",1 DM-1775,"Setup work environment / test builds for refactored repositories","After the repository was split into firefly and new-ife, it was necessary to understand the changes, check for inconsistencies, test builds, set up new IDEA project, etc. ",3 DM-1776,"Obtain and use catalog dd (data definition)","Get catalog metadata, which should include column description, units, and type. Use it in tool tips and possibly to create flexible constraints. A temporary solution is to get metadata from an internal lsst_schema_browser_S12_lsstsim database on lsst-db.ncsa.illinois.edu A permanent solution would be querying catalog metadata from metadata store.",10 DM-1777,"XYPlotter should be caching and restoring plot metadata","For efficiency, XYPlotter is designed to create up to 4 cards. When the card limit is exceeded, a previously created card is reused to plot catalog data for the current catalog. When card is reused the previous plot metadata (like column selections, grid option, etc.) are lost.",4 DM-1778,"Remove optimisation flag management in partition SConstruct","- eups build with -g and -O3 by default. - Developpers can build their sources with next options: scons debug=false opt=3 - Nevetheless partition SConstruct add -O2 option if debug is disabled, which is orthogonal. ",2 DM-1782,"The existing FITS reader class needs to be refactored to improve the performance(2)","- FITS reader class refactoring, improve readability - validation of the new code against the old one - unit test - performance improvement recording if possible",16 DM-1783,"fix faint source and minimum-radius problems in Kron photometry","This transfers some improvements to the Kron photometry from the HSC side: - HSC-983: address failures on faint sources - HSC-989: fix the minimum radius - HSC-865: switch to determinant radius instead of semimajor axis - HSC-962: bad radius flag was not being used - HSC-121: fix scaling in forced photometry The story points estimate here is 50% of the actual effort, as the work (already done) also benefited HSC.",5 DM-1784,"Fix errors in parsing or rendering nested expressions","Qserv has errors rendering nested expressions in predicates of the WHERE clause. It is unclear whether the problem is in constructing the predicate representation or in rendering the representation (or both). Example: {code} SELECT o1.objectId FROM Object o1 WHERE ABS( (scisql_fluxToAbMag(o1.gFlux_PS)-scisql_fluxToAbMag(o1.rFlux_PS)) - (scisql_fluxToAbMag(o1.gFlux_PS)-scisql_fluxToAbMag(o1.rFlux_PS)) ) < 1 {code} Yields: {code} SELECT o1.objectId FROM LSST.Object_100 AS o1 WHERE ABS((VALUE_EXP FACTOR FUNCTION_SPEC scisql_fluxToAbMag(VALUE_EXP FACTOR COLUMN_REF o1.gFlux_PS)-FACTOR FUNCTION_SPEC scisql_fluxToAbMag(VALUE_EXP FACTOR COLUMN_REF o1.rFlux_PS)))<1 {code} I probably left out a more general implementation one/both of those parts of query parsing/analysis.",4 DM-1785,"Add rotAngle to baseline schema","Add ""rotAngle DOUBLE"" to every table that has image ra/decl. ",1 DM-1786,"Implement using multiple disk spindles","Qserv should be able to take advantage of multiple disk spindles (JBOD-type architecture). In practice that means either relying on something like native mysql partitioning, or tweaking loader so that it can distribute chunks across multiple disks (and patch symlinks in mysql data_dir). ",20 DM-1787,"OpenStack automation via Python scripts : Launch an Instance","In our introductory work with OpenStack we have been utilizing the Horizon GUI interface for first steps, followed by the use of command line tools (the 'CLI') (e.g., nova, cinder, etc) as shown in DM-1334 DM-1700 , DM-1701. While it is possible to write automation scripts that utilize the CLI, an approach based on 'pure' Python scripting would fit more seamlessly into the LSST software development process. Enabling OpenStack automation via Python offers the opportunity to integrate provisioning of resources into the overall flow of LSST workflow & processing (e.g., DRP.) The OpenStack services expose native Python APIs that expose the same feature set as the command-line tools. The required python packages (python-keystoneclient, python-novaclient, ..) are installed on the head node 'vlad-mgmt' of the NCSA ISL OpenStack, and so initial Python scripting can be executed/tested there. A first Python script will perform required authentication and launch an instance. ",4 DM-1788,"OpenStack automation via Python scripts : Software installation/test on an LSST node","The set of packages that will enable us to write against the native Python APIs of the OpenStack services is {code} python-keystoneclient python-glanceclient python-novaclient python-quantumclient python-cinderclient python-swiftclient {code} We begin testing these in DM-1787 on the NCSA OpenStack head node, but eventual use within LSST orchestrating workflow would entail these being installed on LSST nodes in the LSST stack. In this issue we perform a basic installation of these packages into the system space on an LSST node/VM for testing. These are managed in github, and we install these via 'pip install' onto an LSST VM for initial tests.",4 DM-1789,"Experiment with afwtable, meas_base, pipe_{tasks, base}",NULL,4 DM-1790,"Produce detailed prototype & accompanying documentation",NULL,4 DM-1791,"Produce straw-man prototype",NULL,3 DM-1792,"Update documentation and automatic install script w.r.t. new newinstall.sh script","newinstall.sh script has evolved and breaks Qserv install procedure.",1 DM-1794,"Pull distEst package into obs_subaru","Reducing HSC data requires an estimate of the distortion, which is provided by the HSC package distEst. This can be pulled into obs_subaru to consolidate code and reduce dependencies. I propose to treat distEst as legacy code, which means I will pull it into obs_subaru without major changes to the code style.",6 DM-1795,"Use anonymous NCSA rsync server to distribute large test datafiles.","DM-1755 has been done before this feature was available. It uses rsync over ssh which require use to have an ssh-key on lsst-dev. NSCA rsync server can now be accessed with next syntax: {code:bash} rsync -av lsst-rsync.ncsa.illinois.edu::qserv/qserv_testdata/datasets/case04/data/DeepSource.csv.gz . # LIST FILES AVAILABLE IN THE MODULE NAMED ""qserv"" rsync lsst-rsync.ncsa.illinois.edu::qserv # To add content to this module/group, you can copy files into the following path: /lsst/rsync/qserv/ {code} rsync over ssh feature should be kept, in order to distribute private data are distributed.",4 DM-1796,"Improve test coverage for case04","Most of queries used in GB-sized case04 return empty results. These queries should be added: {code:bash} fjammes@lsst-db2:~/src/qserv_testdata (u/fjammes/DM-1755) $ mysql --host 127.0.0.1 --port 4040 --user qsmaster qservTest_caseSUI_qserv -e ""SELECT count(*) FROM DeepForcedSource"" +----------------+ | SUM(QS1_COUNT) | +----------------+ | 33349940 | +----------------+ fjammes@lsst-db2:~/src/qserv_testdata (u/fjammes/DM-1755) $ mysql --host 127.0.0.1 --port 4040 --user qsmaster qservTest_caseSUI_qserv -e ""SELECT * FROM DeepForcedSource LIMIT 1"" {code} But, even better, SUI team could provide some more interesting query to Qserv team in order to improve case04 quality.",3 DM-1797,"Package flask","The Data Access Webservice APIs are relying on flask, so we need to package flask according to the LSST standards. For my initial testing, I just run ""sudo aptitude install python-flask"". ",1 DM-1798,"Regression testing of AP Simulator","Run the AP simulator to make sure that none of the changes to the ctrl_events package break anything.",1 DM-1799,"Regression testing of Orca","Do some test runs using Orca to make sure Orca still works after the the changes to ctrl_events.",1 DM-1802,"remove unused local typedefs","gcc 4.8 now warns about locally-defined typedefs that aren't used. We have a few of these in ndarray and afw::gpu that should be removed.",1 DM-1803,"S15 Explore Qserv Authorization","Explore authorization centrally: use information generated by parser. Either generate dummy query and run on mysql that runs near czar, or use info produced by parser to determine if user is authorized. Note, we want to limit this to ~1 week, just to reveal potential problems, or do a quick proof of concept.",8 DM-1804,"Study the current SUI requirement ","Study the current requirement carefully to make sure they all make sense and we can do it.",4 DM-1805,"Study the current SUI requirement ",NULL,4 DM-1806,"Study the current SUI requirement",NULL,2 DM-1807,"Study the LSST data products document and give a summary to the team",NULL,6 DM-1808,"Study the current SUI requirement",NULL,2 DM-1809,"Study SUI requirement and summarize all the input from other team memebers",NULL,10 DM-1810,"segfaults in ip_diffim on gcc 4.8","I'm seeing test segfaults in ip_diffim on gcc 4.8, similar to those resolved on DM-1725, but with no similar smoking gun yet. Preliminary indication is that the problem is actually in meas_algorithms.",2 DM-1811,"Prepare initial content","Prepare LSE-130 content as far as possible without input from the new collimated-projector calibration plan.",4 DM-1812,"Determine LSE-130 impact of collimated projector calibration plan","During a working meeting with Robert Lupton and Chris Stubbs, determine the impact on LSE-130 of the introduction of the collimated projector for calibration.",8 DM-1813,"Prepare draft of LSE-130 for Camera and CCB review","Produce a reviewable draft of LSE-130 based on decisions on calibration operations",4 DM-1814,"Support Camera CD-2 (mainly re: LSE-130)","Provide slides and other information needed for CD-2, mainly relative to the open questions around LSE-130",2 DM-1815,"Support LSE-130 review by CCB (mainly Camera)","Respond to comments, perform revisions to LSE-130 as necessary based on feedback from CCB review of the document",4 DM-1816,"Convert LSE-130 to SysML","Following CCB recommendation of approval of LSE-130 draft, convert Word draft to SysML and provide a docgen to Robert McKercher for final posting. ",2 DM-1817,"Create and post docgen of LSE-68","To support discussions with the Camera, post a provisional docgen of LSE-68 to the appropriate Confluence page. Use knowledge from EA training to improve template.",4 DM-1818,"Support completion of final document","Based on CCB approval of LSE-72 on 10 October, support the completion of the final copy of the document for posting on Docushare.",1 DM-1819,"Complete LSE-140 work as needed to produce final document","Complete any review-driven revisions of LSE-140 and support the CCB meeting and following final document preparation.",2 DM-1820,"LSE-140: Collect desired changes for future release","Prepare for a future revision (Phase 3) of LSE-140. Collect issues to be addressed in the revision. Determine if any affect Phase 2 scope (which would require a prompt revision). It is not anticipated that there will be an actual revision of LSE-140 during the Winter 2015 cycle, because additional detail on calibration requirements will not be available in time.",1 DM-1822,"Fix czar assertion failure","Reported by Tatiana: I am encountering this once in a while. qserv-czar.log python: build/rproc/ProtoRowBuffer.cc:69: int lsst::qserv::rproc::escapeString(Iter, CIter, CIter) [with Iter = __gnu_cxx::__normal_iterator > >, CIter = __gnu_cxx::__normal_iterator, std::allocator > >]: Assertion `srcBegin != srcEnd' failed. Czar is dead and qserv stops responding after that. ---- For more details, search in qserv-l archives mails with ""czar assertion failure"" subject: https://listserv.slac.stanford.edu/cgi-bin/wa for complete description.",4 DM-1824,"Define issues to be addressed","Work with TCS contacts (Jacques Sebag, Paul Lotz, etc.) to define the principal issues",1 DM-1825,"Produce draft of LSE-75 with agreed revisions","Produce a draft of LSE-75 with the following agreed revisions: * remove reference to advance notice of pointing, now in LSE-72 * add reference to PSF reporting",1 DM-1826,"Catch-all epic for essential fixes during DM-W15-1,2,3 ",NULL,10 DM-1828,"Create primary calibration plugins",NULL,9 DM-1829,"Develop General Acceptable Use Policy","Don Petravick and Lee LeClair",6 DM-1830,"Develop Information Classification Policy","Don Petravick, Lee LeClair",6 DM-1831,"Develop Incident Response Policy","Don Petravick, Lee LeClair",6 DM-1832,"Develop Data Management Sub-Project Plan and Risk Table","Don Petravick, Frossie Economou",6 DM-1833,"Develop PMO Sub-Project Plan and Risk Table","Lee LeClair, Iain Goodenow",6 DM-1834,"Develop EPO Sub-Project Plan and Risk Table","Don Petravick, Frossie Economou",6 DM-1835,"Develop Camera Sub-Project Plan and Risk Table","Don Petravick, Richard Dubois",6 DM-1836,"Develop Telescope and Site Sub-Project Plan and Risk Table","Lee LeClair, German Schumacher",6 DM-1837,"Package ISP Documents into LSST Standard Format for Control and Delivery","Robert McKercher",4 DM-1839,"Deblend post-merge objects",NULL,4 DM-1840,"Task-level processing for merged objects",NULL,5 DM-1841,"Fix query error on case03: ""SELECT scienceCcdExposureId FROM Science_Ccd_Exposure_Metadata"" ","Xrootd prevents the worker to return more than 2MB data. On GB-sized data: {code} mysql --host=127.0.0.1 --port=4040 --user=qsmaster --batch -e ""SELECT scienceCcdExposureId FROM Science_Ccd_Exposure_Metadata"" ERROR 4120 (Proxy) at line 1: Error during execution: -1 Ref=1 Resource(/chk/qservTest_case03_qserv/1234567890): 20150123-16:27:45, Error merging result, 1420, Result message MD5 mismatch (-1) {code} On integration test case 04: {code} qserv@clrinfoport09:~/src/qserv (u/fjammes/DM-1841 *)⟫ mysql --host=127.0.0.1 --port=4040 --user=qsmaster qservTest_case04_qserv -e ""SELECT * FROM DeepForcedSource"" ERROR 4120 (Proxy) at line 1: Error during execution: -1 Ref=1 Resource(/chk/qservTest_case04_qserv/6970): 20150204-16:23:43, Error merging result, 1420, Result message MD5 mismatch Ref=2 Resource(/chk/qservTest_case04_qserv/7138): 20150204-16:23:43, Error merging result, 1420, Result message MD5 mismatch Ref=3 (-1) {code}",5 DM-1843,"Permit PropertySets to be represented in event payloads","In the old marshalling code, property sets were representable within the payload of the event. This was removed in the new marshalling scheme. There are things (ctrl_orca) that still used this, so this needs to be added to the new marshaling code. At the same time, new new filtering code can not allow this to be added, because the JMS headers only take simple data types.",2 DM-1844,"Test Qserv on SL7","Needed to run Qserv on CC-IN2P3 cluster.",2 DM-1845,"Coordinate implementation of web form for collecting data about existing data sets","The form is being implemented by the DataCat team. Need to coordinate (including with the NCSA team which parts are covered by which team), test, fine tune etc.",4 DM-1847,"Add support for large results in XrdSsiRequest::GetResponseData","GetResponseData needs to handle data sets beyond 2 MB. The problem is discussed in more details in story DM-1841",12 DM-1848,"SUI work with DB team to define the image query API","IPAC SUI team will work with SLAC database team to define the image query APIs. IPAC needs to make sure the APIs are sufficient to satisfy the UI needs. SLAC will implement them. .",10 DM-1849,"Study RESTful API and work with SLAC team to define image query APIs",NULL,2 DM-1850,"Discuss, review, and define image query APIs with SLAC team",NULL,2 DM-1851,"Image query API discussion and review",NULL,2 DM-1852,"Image query API discussion ",NULL,1 DM-1853,"Image query API discussion",NULL,1 DM-1854,"SUI propose a structure definition for user workspace","Workspace is an integral part of SUI. We want to start the discussion and definition of workspace concept and structure. SUI team had several discussions and Xiuqin presented the results at the DM AHM at SLAC. The slides and the discussion notes are here: https://confluence.lsstcorp.org/display/DM/Workspace+discussion",20 DM-1855,"Specify mechanism for periodic (nightly/weekly) build distribution",NULL,3 DM-1856,"Implement nightly/weekly release automatic distribution - Part I","This ticket covers code in sqre-codekit to do migrate as much of the process of special machines, and git tag repos on the basis of eupspkg manifests.",4 DM-1857,"Github Transition Plan: Write document for CCB",NULL,7 DM-1858,"Release engineering W15 bucket","Bucket epic for activities related to stack releases during W15",24 DM-1859,"Publish v10_0 release",NULL,12 DM-1860,"Update documentation for v10_0 release","All done bar obtaining some release notes. ",2 DM-1861,"Workflow tool improvements w15 bucket","Bucket epic for w15 for improvements with JIRA, Hipchat, etc ",12 DM-1862,"JIRA project for RFCs",NULL,3 DM-1864,"Implement Github transition plan",NULL,10 DM-1865,"Review existing Level 3 documentation","Review existing requirements in this area. Find all relevant existing project-controlled and other key documents.",4 DM-1866,"Document as-is Level 3 requirements and conceptual design","Produce a single jumping-off point for documentation on all aspects of Level 3, on Confluence. Ensure that flowdown for existing Level 3 requirements in SysML is modeled. Describe the high-level conceptual design.",4 DM-1867,"tighten control over heterogeneous DictFields","DM-1218 added support for DictFields with heterogeneous item types, which probably allows a bit too much freedom (the rest of pex_config is much more strongly-typed). Instead of passing None to allow any type to be used, we should pass a tuple of supported types.",1 DM-1868,"Define JSON Results for Data Access Services","As discussed at [Data Access Hangout 2015-02-23|https://confluence.lsstcorp.org/display/DM/Data+Access+Hangout+2015-02-23], we should support json format. This story covers defining structure of JSON results for Data Access Services (dbserv, imgserv, metaserv) ",3 DM-1871,"SUI requirement refinement to define many unclear areas","In the current SUI requirement document, many areas are not clearly defined. We want to put more description and definition for those areas, and identify and define the missing functions. The goal is to generate a requirement document for DM review and put it under version control. ",50 DM-1872,"SUI User workspace specification","At the 2015 Deb DM AHM at SLAC, SUI led a discussion of workspace. https://confluence.lsstcorp.org/display/DM/Workspace+discussion We want to continue the discussion, understand the user needs, identify the DM groups involved. The goal is to generate a document to capture the functions requirement of workspace. The first version of document is here https://confluence.lsstcorp.org/pages/viewpage.action?pageId=41783931",30 DM-1873,"SUI 2D data visualization (XY plot)","Better algorithm in spatial binning to visualize large number of catalog sources Plot histogram for tabular data Plot basic light curve ",40 DM-1874,"SUI the alert subscription system specification","Identify parties involved in the alert system generation, broadcast, subscription. Understand the flow of the alert from generation to notifying users. Understand the requirement for SUI subsystem ""Alert subscription and notification"". ",30 DM-1875,"SUI infrastructure implementation","Identify the hardware resources needed at NCSA for short term development and Set up the basic git repository and build system Explore multi resolution images display for background iamge",40 DM-1876,"SUI functional design ","understand current use cases, collect and define more use cases. Design the major functions of major components in SUI, mainly Firefly package.",60 DM-1877,"SUI web interface and Python interaction","To facilitate users to interact with Firefly visualization components in IPython notebook, to allow users to control the display with Python script. ",60 DM-1878,"Collect, understand, and define more use cases","This is an on-going effort. The collected use cases will be posted at confluence page https://confluence.lsstcorp.org/pages/viewpage.action?pageId=41784036. ",20 DM-1880,"Implement RESTful interfaces for Database (GET)","Implement RESTful interfaces for Database (see all D* in https://confluence.lsstcorp.org/display/DM/API), based on the first prototype developed through DM-1695. The work includes adding support for returning appropriately formatted results (support the most common formats). This covers ""GET"" type requests only, ""POST"" will be handled separately.",5 DM-1881,"Improvements to web form ",NULL,5 DM-1883,"organize the workspace discussion and present a good proposal","SUI team had several discussions and Xiuqin presented the results at the DM AHM at SLAC. The slides and the discussion notes are here: https://confluence.lsstcorp.org/display/DM/Workspace+discussion",10 DM-1884,"summarize WEBDAV capabilities and past experience using it","WEBDAV could be a candidate for managing the user workspace. summarize its capabilities and past experience, collect some use cases will help us to make a better decision.",4 DM-1885,"Contribute to the workspace capability discussion ","This include past experience, collection of use cases. ",2 DM-1887,"HDF5 file format study","Xiquin, Loi, Trey, and myself discussed HDF5 as a default format to return result set and metadata from lower-level database services vs. traditional IPAC table. Here is the summary: Advantages of IPAC Table format - Simple and human-readable, contains a single table - Fixed length rows (easy to page through) - Supported by many astronomical tools - Provides a way to pass data type, units, and null values in the header - More metadata can be added through keywords (attributes) Disadvantages of IPAC table format - Steaming can not be started before all data are received – need to know column width before the table can be written (csv is better alternative) - Only alpha-numeric and '_' characters are allowed in column names (small subset of available characters) - Only predefined datatypes and one attribute type (string) - ASCII representation requires about twice as much storage to represent floating-point number data than the binary equivalent. Advantages of HDF5 - Can represent complex data and metadata (according to LOFAR, good to represent time series) - Structured data, arbitrary attribute types, datatypes can be combined to create structured datatypes - Flexible datatypes: can be enumerations, bit strings, pointers, composite datatypes, custom atomic datatypes - Access time and storage space optimizations - Partial I/O: “Chunked” data for faster access - Supports parallel I/O (reading and writing) - Built-in compression (GNU zlib, but can be replaced with others) - Existing inspection and visualization tools (HDFView, MATLAB, etc.) Disadvantages of HDF5 - Complex - Tuned to do efficient I/O and storage for ""big"" data (hundreds of megabytes and more), not efficient for small reads/writes. - Requires native libraries (available in prepackaged jars, see below) - Not human readable - (?) Not yet widely supported by astronomical tools (counter-examples: AstroPy, IDL, more at hdfgroup site) Tools and Java wrappers: * JHI5 - the low level JNI wrappers: very flexible, but also quite tedious to use. * Java HDF object package - a high-level interface based on JHI5. * HDFView - a Java-based viewer application based on the Java HDF object package. * JHDF5 - a high-level interface building on the JHI5 layer which provides most of the functionality of HDF5 to Java. The API has a shallow learning curve and hides most of the house-keeping work from the developer. You can run the Java HDF object package (and HDFView) on the JHI5 interface that is part of JHDF5, so the two APIs can co-exist within one Java program. (from StackOverflow answer, 2012) * NetCDF-Java is a Pure Java Library, that reads HDF5. However, it's hard to keep pure java version up-to-date with the standard, does not support all the features. A way to set up native libraries (3rd option from JHDF5 FAQ): ""Use a library packaged in a jar file and provided as a resource (by putting the jar file on the class path). Internally this uses the same directory structure as method 2., but packaged in a jar file so you don't have to care about it. Jar files with the appropriate structure are cisd-jhdf5-batteries_included.jar and lib/nativejar/.jar (one file for each platform). This is the simplest way to use the library."" ",1 DM-1889,"S15 Butler (v3)","Improvements and tweaks to the butler as needed.",19 DM-1890,"Research support for integrated view of all databases","Metadata Store needs to contain information about all databases (data release databases, Level 1, Level 3 user databases).",11 DM-1891,"Add support for IPAC table format","Implement support for result formatting in IPAC table format.",6 DM-1893,"Research supporting cutout from images with overlaps","Research producing cutout from images with overlaps.",10 DM-1894,"SUI refactor Firefly package","The team has studied and researched the web UI framework (DM-1148) in W15. A new framework will be decided in February 2015. Many classes in Firefly package need to be refactored to use the the new framework. This epic in S15 will be our first attempt for it. Developers will be involved in this effort: Trey Roby, Loi Ly, Tatiana Goldina, Lijun Zhang, Xiuqin Wu",100 DM-1896,"Design CSS schema to support table deletion","Table/chunk deletion can be an extended process as some worker nodes may be temporarily down. We need to define a process and its supporting structures in CSS to allow gradual deletion of individual chunks and full tables. Deliverable: a design of a system capable of deleting a distributed table (all chunks, all replicas). It should be possible to create a table with the same name after deletion.",4 DM-1897,"Modify CSS structure to support table deletion","Modify CSS structures to support DROP TABLE, as defined in DM-1896.",2 DM-1898,"Consistency checking for table data CSS ","CSS data on tables/chunks/nodes is supposed to be consistent at all times. Would be nice to have a tool that verifies consistency, probably including checking actual worker state.",4 DM-1899,"Tool to dump CSS information","CSS information tree may become large and it would be nice to have a tool to examine that tree or parts of it. Something that dumps the tree in user-friendly way and allows filtering or summarizing.",2 DM-1900,"Worker management service - design","We need to replace direct worker-mysql communication and other administrative channels with a special service which will control all worker communication. Some light-weight service running alongside other worker servers, probably HTTP-based. Data loading, start/stop should be handled by this service.",5 DM-1901,"Re-implement data loading scripts based on new worker control service","Once we have new service that controls worker communication we'll need to reimplement WorkerAdmin class based on that.",8 DM-1903,"Implementation of calibration transformation framework","Following DM-1598 there will be a detailed design and prototype implementation for the calibration & ingest system. This issue covers cleaning up that code, documenting it, having it reviewed, and merging to master.",2 DM-1904,"Continued footprint improvements","A redesigned API and support for topological operations within the Footprint class. This continues the work started in DM-1107 in W15. Breakdown: jbosch 15%; swinbank 85%",8 DM-1905,"QSERV issues when working with scisql_s2PtInCircle","This problem has been adressed un u/fjammes/DM-1841. Here's Tatiana report: {quote} This error happens on all DeepForcedSource queries. (It happens on DeepSource too, but not always.) [2015-01-15 11:02:29] [Proxy][4120] Error during execution: -1 Ref=1 Resource(/chk/LSST/6970): 20150115-11:01:05, Complete (success), 0, Ref=2 Resource(/chk/LSST/7138): 20150115-11:02:19, Complete (success), 0, Ref=3 Resource(/chk/LSST/7140): 20150115-11:02:18, Error merging result, 0, Ref=4 Resource(/chk/LSST/730 (-1) Query examples: select * from DeepForcedSource where ra>0.4 and ra<0.6 and decl>0.9 and decl<1.1; select * from DeepForcedSource where scisql_s2PtInCircle(ra, decl, 0.5, 1.1, 0.138) = 1; {quote}",2 DM-1906,"Outline an expandable Python framework for advanced users ","Outline an expandable Python framework for use by advanced users in interactive or batch mode. Many users should be able to contribute software to the framework following simple API guidelines. Also look at configuration and delivery systems for the software. Start by looking for existing Python software that could be used in the areas of: - Database & file access - Data analysis frameworks - Display of data,, especially in an Astronomical context. - Reading and writing data in different formats - Graphing data locally or over the web - Science and astronomy data analysis - Event interfaces - VO functionality and connecting with existing software like DS9,, Aladin, Topcat, etc. - Ways to integrate modules with an LSST focus like AGN Large Scale Structures Galaxies Local Volume Solar System Astrostatistics Stellar Pops Strong Lensing Supernova Transients Weak Lensing Camera - Configuration and delivery system ",12 DM-1907,"Backport HSC multi-band deblend processing","Breakdown: lauren 60%; price 40% We need to transfer the recent HSC-side multi-band deblender changes to the LSST side, including all HSC issues in this query: https://hsc-jira.astro.princeton.edu/jira/issues/?filter=11603",35 DM-1909,"Interface design for full focal plane PSF estimation","Breakdown: swinbank 40%; jbosch 30%; rhl 30% This is mostly design work, but I'd like the goal to be a new Python command-line task that repeats as much of the current ProcessCcd as necessary to run full focal plane PSF estimation, and would serve as a starting point for a future-proof visit processing script. Some issues include: - Gather requirements from e.g. DESC people as to needed inputs and data flow. - Get details of what camera/telescope systems will provide, and figure out how those related to what DESC needs. - Design classes for camera/telescope engineering data and wavefront information. Figure out how they'll be managed on disk (stored with Exposure, part of CameraGeom, with flats, biases, etc.). - Discuss parallelization needs with middleware team, and determine a way forward that lets us design interfaces using future parallelization schemes that don't exist yet. - Design Python task interface for full focal plane PSF estimation. - Implement command-line task that calls the full focal plane PSF estimation task. - Implement placeholder PSF estimation subtask that just uses existing PSF determiners on single CCDs. This may be too ambitious for the 40 story points we've allocated, and should discuss either adding more effort or reducing the scope.",45 DM-1910,"DRP DM-S15-1,2,3 Bugs and Papercuts","Breakdown: jbosch 16%; lauren 16%; rhl 20%; pgee 16%; price 16%; swinbank 16%",20 DM-1912,"DRP DM-S15-4,5,6 Bugs and Papercuts","Breakdown: jbosch 16%; lauren 16%; rhl 20%; pgee 16%; price 16%; swinbank 16%",20 DM-1913,"Prototype command interaction with Firefly",NULL,10 DM-1914,"Research Javascript Frameworks: Finish new framework proposal",NULL,18 DM-1915,"Create a deployable installation package for Firefly","With Firefly being open-source, we should provide a simple all-in-one installation package so a user can quickly setup and deploy an instance the Firefly Tools web application.",10 DM-1916,"Fine-tune data access interfaces","Brought up by Gregory via comments on the API page. Data release selection in queries: I see that the /db/... queries take a ""?"" query parameter ""db"" with an example value of ""DR1"", i.e., a data release selector. A couple of remarks: * Will this query parameter be provided for all the Level 2 image data products, e.g., for retrievals of coadded images? **If so, then it needs an equivalent to the M4 ""GET /meta/v0/db/"" query. * I assume the ""db"" query parameter defaults to the most recent data release. * Will the M4 query return an indication of which ""?db="" value is the current default? * I assume that the numeric-identifier components of the various paths are unique only within a single data release. That means that eventually, in user documentation, we should make sure that they understand that they can't scan through different releases' versions of the same image (for example) just by varying the ""?db="" parameter. * Are the identifiers also unique within a particular type (i.e., ""raw"", ""template"", ""coadd"", ""calexp"", etc.)? ----- Distinguishing L1 and L2 versions of reprocessed data products Since most or all of the L1 data products will be regenerated in each data release, the catalog and image APIs should presumably allow the user to distinguish between the two. I see how this could be done for catalogs - the ""?db="" parameter presumably allows selecting something like ""L1"" (for the actively updated Level 1 database) in addition to the above-documented ""DR1"", ""DR2"", etc. Will the L2 table names for the reprocessed L1 data products be generally expected to be the same as for L1? (Barring the discovery of a serious issue that requires revision of the schema for the reprocessing.) How will the L1 and reprocessed-L1 image data products be distinguished?",9 DM-1917,"Fix missing virtual destructors","The compiler is warning about some derived class hierarchies that are lacking virtual destructors. We should add at least empty implementations to the base classes of these hierarchies.",1 DM-1918,"write Unit test and validation classes to validate the FITSreader refactoring ",NULL,8 DM-1919,"Address misc. compiler warnings","Fix places where compiler is warning about some things we are doing on purpose and which we don't intend to change. This helps keep compiler noise down so its easier to notice ""real"" warnings.",1 DM-1920,"update shapeHSM wrappers to latest external version","The HSM shape code has undergone many improvements and bug fixes as part of being included in the GalSim package, and we've recently included those in the HSC fork of meas_extensions_shapeHSM (HSC-129, HSC-1093). We should transfer those changes to the LSST side before tackling DM-981 (or at least before finishing it). The story point estimate here is for the work already done on the HSC side (with the usual 50% factor for shared work). The transfer to the LSST side should be essentially no effort. To the extent that EVM cares about this, the credit should go to [~price], even though I ([~jbosch]) am doing the transfer.",6 DM-1921,"Make unit tests use shared libraries","Many (all?) unit tests are currently built as static executables which include all needed object files. This has several issues associated with it: - many files are compiled twice, once as *.os files for shared libraries, second time as *.o file for unit tests - unit tests do not test actual code in the shared libraries but instead separately-built copy of the same code We should change our procedure and make unit test to link against shared libraries to avoid these problems.",4 DM-1923,"Setup network for IPMI",NULL,2 DM-1924,"Setup IPMI bastion hosts",NULL,5 DM-1925,"Document how to use IPMI with LSST infrastructure",NULL,1 DM-1926,"Base configuration of NFS servers","install and configure OS",3 DM-1927,"Test new NFS servers","Test to confirm servers are configured optimally",15 DM-1928,"Deploy first of the NFS servers",NULL,20 DM-1929,"LOE - Week ending 12/26/14",NULL,4 DM-1930,"LOE - Week ending 1/9/15",NULL,8 DM-1931,"LOE - Week ending 1/16/15",NULL,4 DM-1932,"LOE - Week ending 1/23/15",NULL,8 DM-1934,"LOE - Week ending 2/6/15",NULL,8 DM-1935,"LOE - Week ending 2/13/15",NULL,8 DM-1936,"LOE - Week ending 2/20/15",NULL,8 DM-1937,"LOE - Week ending 2/27/15",NULL,8 DM-1939,"Define instrumental inputs to PSF estimation","Define inputs needed to build physically-motivated PSF models beyond what's contained in the image data and the current CameraGeom. This includes: - static engineering data from lab tests - slowly-varying engineering data (measured daily, weekly, etc.) - per-visit auxiliary data from telescope and camera (including, but not limited to wavefront sensor data). The focus here should be on APIs, not the technical details of the data; we want to define class hierarchies (perhaps some polymorphic ones) that can be used to pass this data around in the future. We also want to identify and characterize any preprocessing that needs to be done before they can be used for PSF estimation. Experts who should be consulted include [~gpdf] (who is in charge of making sure the camera and telescope interfaces to DM are well-defined), camera/telescope people he recommends, and people from the LSST DESC who have worked on physical PSF estimation and know something about the things they'll want (e.g. Michael Schneider, Aaron Roodman). This is essentially a requirements-gathering task, so the output should be a confluence page, not code.",10 DM-1940,"API for instrumental inputs to PSF estimation","Following the requirements-gathering in DM-1939, come up with classes that can be used to pass the needed inputs to the PSF estimation code, and at least sketch out roughly how they will need to be managed by the butler and loaded by the framework code. The output of this ticket should be an API design in confluence (possibly on the same page used for DM-1939), and an associated RFC.",10 DM-1941,"Organize SUI design discussions","Organize the SUI team for web UI discussions to capture as much functions and components as possible. A draft design document should be produced out of those discussions. ",10 DM-1942,"Test data development and HSC stack integration","Breakdown: price 20%; lauren 80% This epic is focused on general stack testing and integration on HSC data, with a goal of getting LSST-side reductions of HSC data to the same level of quality and robustness currently present on the HSC fork of the stack, which will involve a combination of backporting minor fixes from the HSC codebase and tuning parameters for LSST-side algorithms that supercede their HSC-side counterparts. This will allow science-level algorithms to be tested using HSC data, and will provide functionality important for science-grade tests on other real data (such as the ongoing CFHTLS reprocsessing at IN2P3). This effort will be focused on single-epoch processing (i.e. ProcessCcdTask), with coadd-level processing a stretch goal dependent somewhat on the completion of astrometric calibration work at UW.",45 DM-1943,"HSC backport: convert Peak to PeakRecord","This issue covers transferring all changesets from [HSC-1074|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1074] and its subtasks, as well as: - An RFC to propose the API change, and any requested modifications generated by the RFC. - Additional fixes to downstream code that's broken by this change (HSC-side changesets should be present for most of downstream fixes, but perhaps not all).",8 DM-1944,"HSC backport: guarantee consistent handling of peaks in deblender","This issue covers transferring changesets from: - [HSC-134|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-134] - [HSC-1109|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1109] - [HSC-1083|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1083] ",4 DM-1945,"HSC backport: multiband processing for coadds","This issue includes transferring changesets from many HSC issues: - HSC-1060 - HSC-1064 - HSC-1065 - HSC-1061 Most of this is in multiBand.py in pipe_tasks, but there are scattered changes elsewhere (including updates to camera mappers to include the new datasets, for which we'll need to modify more than just obs_subaru). However, before we make these changes, we'll need to open an RFC to gather comments on the design of this task. We should qualify there that this is not a long-term plan for consistent multiband processing (which we'll be starting to design on DM-1908), but a step towards better processing in the interim. Note: while I've assigned this to [~lauren], as I think it will be very helpful for her to get familiar with this code by doing the transfers, the RFC will have to involve a collaboration with [~jbosch], [~price], and Bob Armstrong, as we can't expect someone who wasn't involved in the design to be able to write a document justifying it.",8 DM-1946,"HSC backport: low-level Footprint merge code","This is a transfer of changesets from the follow epics: - [HSC-1020|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1020] - [HSC-1075|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1075]: only the afw changes Because this is purely an addition to the interface, and we're planning to redesign that interface in DM-1904, I don't think we need an RFC here.",4 DM-1947,"Add support for async request cancellation to xrdssi",NULL,12 DM-1948,"S15 Implement Query Cancellation","Add support for query cancellation.",32 DM-1950,"Add abstraction in czar for unit tests","Add hooks in czar that will let us build unit tests.",8 DM-1951,"Unit test for query cancellation",NULL,8 DM-1952,"Change log priority for message ""Unknown column 'whatever' in 'field list'"" ","Next message should be logged with ERROR priority: {code} 0204 15:08:03.748 [0x7f1f4b4f4700] INFO Foreman (build/wdb/QueryAction.cc:250) - [1054] Unknown column 'whatever' in 'field list' {code}",1 DM-1953,"Post meas_base move changes to Kron","These are to note leftovers from DM-982. They could be done in a single issue. 1. I commented code out referring to correctfluxes, but it will need to be restored once it is available in the new framework. 2. Jim asked me to replace the computeSincFlux which is currently in PsfImage.cc in meas_algorithms with a similar call in meas_base/ApertureFlux.cc. I did not do this because it became rather complicated, and can just as easily be done when the meas_algorithms routine is moved or removed. Basically, the templating in ApertureFlux is on Pixel type, whereas in meas_algorithms it is on ImageT (where ImageT is not necessarily a single class hierarchy -- e.g., Image and MaskedImage). So I left this for now.",1 DM-1954,"HSC backport: deblended HeavyFootprints in forced photometry","This is a transfer for changesets for [HSC-1062|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1062]. Unlike most of the HSC backport issues for multiband deblending, these changes will require significant modification the LSST side, because we need to apply them to the new forced measurement framework in meas_base rather than the old, HSC-only one in meas_algorithms and pipe_tasks. Also include [HSC-1256|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1256], [HSC-1218|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1218], [HSC-1235|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1235], [HSC-1216|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1216].",20 DM-1955,"International Network Design and Implementation","FIU/Amlight is expected to provide a full capacity 3 x100 Gbps link by FY17. ",12 DM-1956,"REUNA will provide a “pre-operations” link between La Serena and Santiago","REUNA will provide La Serena - Santiago links at full capacity 100 Gbps link by FY17 and a 40 Gbps secondary link by FY20. This link will support testing and development prior to that time.",14 DM-1957,"Chilean National Network Design and Implementation","Refer to REUNA MREFC sub award contract for deliverable details. This covers non-contract work by the AURA/LSST.",20 DM-1959,"S15 Data Distribution & Replica Mgmt Prototype","This epic covers building an initial prototype of the Data Distribution and Replication system. The design is covered through DM-1060",30 DM-1961,"Prepare data set for large scale tests","Load data set for large scale tests (duplicate, partition, load)",10 DM-1962,"Run large scale tests","Coordinate running large scale tests",6 DM-1964,"Parallelization requirements for PSF estimation","We need to gather algorithmic ideas for how full focal plane PSF estimation will work from a parallelization and data flow standpoint, and discuss with the middleware team how these should be handled from an interface standpoint. Questions include: - Will we need to do significant cross-CCD image processing or require significant memory for these tasks? If so, should we structure this via message passing between CCD-level processes, or scatter-gather? - Assuming a scatter-gather approach, will we need multiple scatter/gather iterations when processing a single visit? - How much data will be passed between threads/processes? Would this include complex serializable objects, or just POD arrays? - Will different PSF estimation plugins will have different parallelization requirements? Or, can we define the plugin interfaces at a low-enough level that parallelization can be handled by the framework? If plugins do need to control their own parallelization, how do we make parallelization interfaces accessible to the plugins?",8 DM-1965,"Python interface for full-visit PSF estimation","Create a Python interface for a pluggable PSF estimation system that supports algorithms that will operate over full images. This should include a sketch of how a calling command-line task, and placeholders for parallelization interfaces that may not yet be finalized. The output of this issue is a completed RFC on the design.",6 DM-1966,"Command-line driver and placeholder implementation for PSF estimation","Create a command-line task that makes use of the new PSF estimation interface, duplicating as much of ProcessCcdTask's functionality as necessary to provide the inputs to PSF estimation (I expect this new task to ultimately replace ProcessCcdTask). This may have to include workarounds or temporary implementations for parallelization features that are not yet available. Create a simple PSF estimation placeholder that simply uses existing single-CCD PSF-determiners.",6 DM-1968,"CModel flux validation and testing","Investigate the performance of the new version of the CModel code on various test datasets, including HSC data (following DM-1942), SDSS, and possibly CFHT data. Breakdown: lauren 100%",35 DM-1969,"Create a kind of Wcs that encapsulates a TAN WCS and a distortion model","We can simplify the astrometry solver if we have a Wcs that encapsulates a pure tangent-plane WCS and a distortion model that takes converts between PIXELS and TAN_PIXELS. This is useful because at the early stages of processing raw data we have a TAN WCS from the telescope control system and a pretty good estimate of distortion from the optical model (represented in the camera geometry). The result will be a Wcs whose sky<->pixel transformation is a reasonable approximation of reality (and a much better approximation than the tangent-plane WCS that is currently available. This will potentially eliminate a significant amount of confusing code that attempts to correct for distortion by manually applying the optical distortion model.",4 DM-1973,"Build 2015_02 Qserv release","See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe.",1 DM-1974,"Fix enclose, escape, and line termination characters in qserv-data-loader","Add this string to mysql loader 'LOAD DATA INFILE' command: {code} q += ""ENCLOSED BY '%s' ESCAPED BY '%s' LINES TERMINATED BY '%s'"" % (enclose, escape, newline) {code} and add params in cfg file.",2 DM-1977,"S15 Implement Fully-RESTful Data Access Web Service","Improvements to the skeleton of Data Access Web Service built in W15. Make all responses fully RESTful (including results, and errors, setting response headers). Fine-tune API, add support for versioning, unit testing.",39 DM-1978,"Research and Document API Versioning","Research and document versioning of the RESTful API (through flask blueprints). In particular, need to understand how to avoid code duplication between different versions of API.",4 DM-1979,"Add SQLite-based v0.1 unit testing for metaserv","Add unit tests for the RESTful flask based API. I think it'd be most useful if we could a) load some test data into underlying metaserv b) run programmatically things like ""curl -H accept:text/html http://127.0.0.1:5000/meta/v0/db/L2/DC_W13_Stripe82/tables"" etc (more examples in dax_*serv/README.txt) and verify that we got what we expected. Do it for both json and html.",4 DM-1980,"Add error handling for webserv","Add basic error handling for the RESTful flask-based API",6 DM-1981,"Improve security for mysql in python","Revisit all code that talks to mysql from python to use parameter bindings instead of direct string substitutions. In practice: {code} conn.execute(""SELECT * FROM t WHERE name=%s"" % theName) {code} should be replaced with {code} conn.execute(""SELECT * FROM t WHERE name=%s"", (theName,)) {code} For details, see http://mysql-python.sourceforge.net/MySQLdb.html#some-examples ",5 DM-1982,"Fix JDBC timestamp error","JDBC driver returns an error on next query: {code:sql} sql> select * from Science_Ccd_Exposure [2015-02-06 13:39:37] 1 row(s) retrieved starting from 0 in 927/970 ms [2015-02-06 13:39:37] [S1009] Cannot convert value '0000-00-00 00:00:00' from column 32 to TIMESTAMP. [2015-02-06 13:39:37] [S1009] Value '[B@548997d1' can not be represented as java.sql.Timestamp {code}",1 DM-1987,"Redesign/Refactor WCS and Coord","%50 KSK, %50 RO Currently WCS is mutable and Coord objects are heavyweight. Refactor WCS to be immutable and make Coord less heavyweight. Include lists of Coord objects. It's possible astropy could inform in that area. Also, remove TanWcs in favor of TanSipWcs since TanWcs can have SIP terms.",40 DM-1988,"Update analysis tasks: diffim and snap combination","50% KSK 50% RO A small amount of this work is in 02C.03.01 The diffim task needs to be looked at. It needs to be updated to use current mechanisms. It should also be refactored to split out some of the procedural code into methods. In a similar task, the lsstSimIsrTask needs to include a real snap combine step. Currently, one of the snaps is dropped on the floor. For this round just implement naive snap addition and morphological CR rejection.",43 DM-1989,"Define API for Stack Astrometric Calibration","70% RO 30% KSK This should also include a minimal implementation. This should be done with an eye toward photometric calibration. Prerequisite: Get multi epoch (multi-band?) catalogs of centroids from some trusted source (CFHT, HSC?). 1. Load all stars that overlap a patch for all epochs. 2. Associate all stars on each chip. 2a. Implement K-D tree 3. Fit model for rigid chip system + optics + atmosphere. Eigen for sparse model fitter. 3a. Allow for external catalog. 3b. This could include a class to fit XYTransforms 4. Turn result into WCS. So maybe a down scope for a single cycle epic is to get matches and interfaces for models and solvers. ",75 DM-1990,"Research DCR in the context of DiffIm including possible algorithms for mitigation.","It is not clear how template coadds will be built. This includes understanding the data necessary to generate a template for the entire sky. This epic is to identify possible techniques as well as the risks associated with each technique. This does not need to pin down the exact algorithm or the specific selections, but should inform what further development is necessary to avoid putting alert generation at risk.",19 DM-1991,"Refactor Approximate and Interpolate classes","100% RO A base class for this will be created in DM-740 as a part of epic DM-85. This epic is to implement the classes to replace the original functionality.",34 DM-1992,"SQuaRE Support","50% KSK 50% RO SQuaRE has asked that we leave 20 SP free per Cycle to help out.",20 DM-1994,"Story point display and roll-up in epic display","I understand that there is a pending request to display the story points for individual story issues in the mini-table in which they are displayed for an epic. It would also be useful to see a rolled-up total of the story points for the defined set of stories - so that, among other things, this could be compared to the story point value for the epic. Ideally the story points for the roll-up might be displayed as ""nn (mm)"" where nn is the total points and mm is the number of points remaining to do (or done already - I don't care which as long as the definition is clear).",1 DM-1996,"Define faulty/consistent states and recovery process","Whiteboard session(s) to gather a list of invariants / principles that define a consistent state, a list of fault conditions, and the steps in failure recovery.",12 DM-1997,"Data transport mechanism for data distribution","Decide on a transport mechanism for data (bit torrent, scp, or ?). We must take into consideration whether the data source matters in this choice (e.g. tape vs known good node), as well as how to identify that a data source is the correct one (e.g. via checksums and sequence numbers).",12 DM-1998,"Architecture for failure detection and resolution","Come up with an architecture for detecting failure or non-nominal conditions (e.g. under replication). The core question to resolve is whether we go with a distributed approach, or with centralized control.",16 DM-1999,"Research existing theory and prior art","Peruse the distributed systems literature for prior approaches to this problem, and examine existing system implementations for components/ideas we could reuse.",10 DM-2000,"Document data distribution/replication plan","Produce an overview document that explains our definitions, architecture and strategies for dealing with data distribution and replication. ",10 DM-2001,"Define strategy for adding and removing data ","Define a strategy (push/pull distributed/decentralized) for recognizing incoming data and cleaning up/removing stale/deleted data. Is data ingest just another form of failure recovery?",10 DM-2002,"Define data distribution/replication testing strategy","Once we decide on a design for data distribution / replication, we should come up with a test plan.",4 DM-2003,"Package Reorganization (Science Pipelines)","Breakdown: jbosch 50%, swinbank 50%",38 DM-2005,"switch ndarray to external package","There is already an external ndarray project on GitHub (we've been using a fork of that). We should merge the forks and switch to using the external package. ",2 DM-2006,"merge ""basics"" packages","Create detailed RFC and implement merge of base, utils, and daf_base. See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning",2 DM-2007,"separate pex_exceptions from base and rename","Remove dependency on base from pex_exceptions and rename to just ""exceptions"" (after RFC). See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning",1 DM-2008,"Move Wcs from afw::image to afw::coord","Create RFC and implement move of Wcs from afw::image to afw::coord. See https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning",1 DM-2010,"move Jarvis/shapelet code to legacy package","Create RFC and remove Jarvis/shapelet package from meas_algorithms, into new legacy sci package. See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning",1 DM-2011,"move Psf, Kernel code to new afw::convolution subpackage","Create detailed RFC and implement move for these packages. See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning",2 DM-2012,"split pipe_base into command-line and non-command-line components","Create detailed RFC and implement package split, to separate basic Tasks (to be used as e.g. subtasks) from CmdLineTask and ArgumentParser. See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning",1 DM-2013,"Implement image stitching","This story involves implementing code that stitches images, simple case that does not involve tract boundaries. More advanced case in covered in separate ticket. We will need to determine WCS information for the target images.",6 DM-2014,"Create interface and utility package for single-frame/forced processing","Create detailed RFC and implement move of interface and utility code from multiple existing packages to new package. See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning",2 DM-2015,"Design and implement RESTful API for image stitching and rotation",NULL,4 DM-2016,"Split PSF estimation and PSF model code into separate package","Create detailed RFC and implement move of concrete PSF estimation code and Psf subclasses from meas_algorithms to separate package. See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning",1 DM-2017,"rename packages with minimal reorganization","meas_deblender, ip_isr, meas_astrom, meas_modelfit, and ip_diffim do not require major refactoring to fit into the new package reorganization, but they should be renamed (with sci_prefixes). See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning",1 DM-2018,"Split measurement plugins into separate packages","Create detailed RFCs and implement splitting measurement plugins into separate package. May want one package for extremely basic plugins, always-used plugins (PixelFlags, TransformedCentroid, etc). See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning",2 DM-2019,"Split coaddition code and single-frame/forced command-line drivers","This should split all content in pipe_tasks into two packages (aside from what may have been removed in previous issues).",2 DM-2020,"Research how to support L3","Research implications of having to deal with updatable Level 3 data.",12 DM-2021,"Architecture for supporting small non-partitioned tables","Some tables, like Exposure, provenance, will not be partitioned, and the current plan is to either replicate them on each node, or store on shared file system. Need to decide how it will be dealt with.",6 DM-2022,"Research software deployment on the qserv cluster","Discuss and decide how to deploy and upgrade Qserv software",10 DM-2023,"Investigate procedures for package reorganization","e.g. develop script to handle bulk namespace changes.",4 DM-2024,"DRP S15 support for SQuaRE","Breakdown: jbosch 16%; lauren 16%; rhl 20%; pgee 16%; price 16%; swinbank 16%",20 DM-2025,"FY15 Key Performance Metrics","Collect data, compile scripts, perform measurements as necessary to report figures in respect of the FY15 key performance metrics. Breakdown: lauren 50%; rhl 50%",16 DM-2026,"Support Exposure Use Cases","Development in support of Exposure Use Cases",19 DM-2027,"Implementing stitching multiple patches across tract boundaries in a coadd",NULL,4 DM-2030,"refactor afw Swig to improve build times","I have an idea for how to improve Swig build times that we should get vetted (and possibly improved upon) by a true Swig expert (even if that costs a bit of $$). This issue includes vetting that idea (splitting up classes into multiple per-package module builds), getting it through the RFC process, and implementing it in afw.",10 DM-2031,"Add image-query related KPIs to the plan","Existing plan in LDM-240 does not mention image related KPIs. Need to come up with a road map, and propose KPIs. This should be synchronized what realistically NCSA cluster can deliver in any given FY.",5 DM-2034,"FY19 Setup Database for Deep Drilling",NULL,53 DM-2035,"FY18 Setup Calibration Database","Need to think through issues related to supporting calibration. Schema, requirements that will require special optimizations.",39 DM-2036,"S17 Build Prototype of AlertProd and L1 User Database","Build a working, non-optimized prototype of the [Alert Production and L1 User Database|http://ldm-135.readthedocs.org/en/master/#alert-production-and-up-to-date-catalog]. Deliverable: working, non-optimized prototype of AlertProd Database.",79 DM-2037,"X16 Revisit Design of AlertProd & L1 Db","Revisit the [design of Alert Production Database and Level 1 User database|http://ldm-135.readthedocs.org/en/master/#alert-production-and-up-to-date-catalog], including schema, indexing, partitioning, synchronization, replicating, fail over. Verify that the latest requirements match the design. This epic will likely involve experimenting and light-weight standalone prototyping related to parts of the system that might be non-trivial to scale or to implement. Deliverable: a refreshed design document for Alert Production and L1 User Database.",19 DM-2038,"FY17 Design Internal DRP Database","Internal DRP DB will be used to store * all bookkeeping (provenance, what run what did not, etc) * intermediate data products (might be larger than final data products), * a subset of data (what we need by DRP), eg foorprints of objects Internal DRP DB might need its own spatial engine. It is expected that SDQA will run on that database. Need to think through issues related to supporting internal Data Release Production. Schema, requirements that will require special optimizations. Need to define reliability requirements. In limited cases pipelines might want to use internal db (instead of files). Example: select all objects from a given region. Need to understand query load and complexity coming from DRP.",79 DM-2039,"FY18 Revisit Design of L3 Support in Qserv","Need to think through issues related to supporting Level 3 databases",100 DM-2040,"FY19 Design Next-to-database Data Analysis System","Need to design the system that will allow users run their own custom data analysis next to database.",100 DM-2041,"FY19 Implement Next-to-database Data Analysis","Need to implement the system that will allow users run their own custom data analysis next to database.",100 DM-2042,"W16 Improve Data Provenance Design","We have a detailed design of the Provenance, described at https://dev.lsstcorp.org/trac/wiki/db/Provenance. Work covered by this epic involves: 1. Revisiting the design and tweaking it as necessary. In particular: * Describing in more details interactions with key data producers (DRP, AlertProd, Calibration, L3 data brought in by users). * Estimating the size of provenance data * Considering querying the provenance data 2. Evaluating existing off-the-shelf provenance systems/tools. Deliverable: a document describing data provenance architecture / schema supported by a standalone proof-of-concept prototype.",69 DM-2044,"Catch all epic for essential fixes in Science Pipelines DM-W15-5",NULL,10 DM-2045,"server side preparation for histogram plot (1)","Convert necessary code to make it possible for a JavaScript component to place a JSON request to the server and to parse the resulting RawDataSet. ",6 DM-2046,"Client side plot display for histogram","- Create a React JavaScript component, which takes the data and renders histogram. - Make it possible to call this component from GWT code, using experimental JsInterop technology in GWT 2.7",10 DM-2047,"SUI Investigate L3 data/tools requirements, evaluate potential tools ","There are many overlap areas in L3 data analysis tools with the general science user tools. We want to identify those requirements and needs to help making SUI components adaptable for L3 data production and analysis.",30 DM-2048,"Start requirements gathering for pipeline QA visualization needs ","Gather use cases for pipeline QA visualization tools. We want to build the SUI components in such a way that they could be used to support QA needs. ",36 DM-2049,"SUI Build the visualization components that could be used independently","Currently we identified three basic components: Image visualizer, tabular data display, 2D XY Plot. All three could share the data model and provide inter activities between the components. ",80 DM-2050," Integration and test monitoring architecture Part I","[retitled to better capture cycle scope] Develop and deploy a layer to capture the outputs, initially numeric, of integration testing afterburners such as sdss_demo, hsc_demo, and others developed this cycle. Also capture meta-information such as execution time and memory footprint. Propose log format to standardise production of such informations. Investigate notification system based on trending away from expected values. Investigate data provisioning of integration tests such as storage of test data in GithubLFS. [75% JMP 25% JH] ",100 DM-2051," Firefly-based data display for SQuaSH - Part I","[Epic retitled to better reflect cycle scope] This epic covers work relating to working on the visualisation side of the Science QA Analysis Harness. It is a timeboxed effort to come up to speed with Firefly in particular, evaluate it against our needs, and provide any feedback to the Firefly team. Some prototyping of visualising integration dataset products will also be involved. [AF 100%] ",45 DM-2052,"Maintain list of OSes that pass build and integration testing ","Provide an automatiically generated and updated pages showing operating systems that are successfully building and integrating the stack from source. [FE at 75%, JH at 75%]",20 DM-2053,"Specify system for performing CI on Docker stack containers","Investigate how we can CI first-party Docker containers with runnable stacks [JH 100%]",30 DM-2054,"Release engineering Part One","Bucket for public stack releases [FE at 75%, JH at 75%]",40 DM-2055,"Miscellaneous service support improvements","JIRA, comm toos, etc for DM and non-DM teams (indicate) In order to avoid fractional story points, some 0 [FE at 75%, JH at 75%]",16 DM-2057,"Attend Scale 13x conference","Attend database talks, in particular the MaxScale proxy talk (http://www.socallinuxexpo.org/scale/13x/presentations/advanced-query-routing-and-proxying-maxscale?utm_campaign=north-american-trade-shows&utm_source=hs_email&utm_medium=email&utm_content=16099082&_hsenc=p2ANqtz-_MFjfxvpCdmV_Ax2RKDdOGypHPQ85UL-UMuy0eRs_MrlJ2qJVp-MXx-g7_-dAQsq0trpA61hkZrzO-3gp6bKVkpK52fQ&_hsmi=16099082). If anyone has questions they would like me to ask, please post them here as well. I will post notes to this issue. ",2 DM-2058,"Data loader should always create overlap tables"," We have discovered that some overlap tables that are supposed to exist were not actually created. It looks like partitioner is not creating overlap files when there is no overlap data and loader is not creating overlap table if there is no input file. Situation is actually symmetric, there could be non-empty overlap table but empty/missing chunk table. When we create one table we should always make another as well. ",2 DM-2059,"Clean up QuerySession-related code in czar","(created in response to DM-211) This ticket should address the following inelegancies in the qserv-czar. * QuerySession->QueryPipeline. The ""session"" abstraction has moved to a better place. The iterator portion should be shifted into its own separate class, though perhaps still associated with QueryPipeline. The iterator portion's new home should be amenable to eventually moving the actual query materialization to the worker, though we shouldn't introduce new abstractions until we are actually ready to move the substitution/materialization to the worker. * QueryContext needs to be split into incoming external QueryContext and a sort of QueryClipboard for passing information between analysis/manipulation plugins. Eventually, I imagine a chain/tree of them attached to the select statements themselves in order to represent subquery scope nesting (which is complicated to represent and to reason about--nesting and the resulting namespace resolution is tricky), but I don't think we should try doing the chaining in the first phase. For this ticket, create QueryClipboard to hold the portion for interchange between analysis plugins. Query analysis plugins would then pass this object (which points at an immutable? QueryContext) between themselves. QueryClipboard probably should live in qana, QueryContext in query (unless there is a good reason to move it). ",8 DM-2060,"Rename TaskMsgFactory2","Rename TaskMsgFactory2 to TaskMsgFactory. Please see DM-211 for more information.",1 DM-2061,"Port fault-recovery testing code to XrdSsi","Please see DM-211 for the origin of this ticket. BillC put in code to introduce random errors in query dispatch as part of working on code to recover from faults. In the port to the XrdSsi API, we did not port this code. This story is to introduce the ability (compile-time configurable, if not command-line or dynamically configurable) to simulate these sorts of faults to exercise the fault-recovery (confined to retrying on transient-ish failures) code.",12 DM-2063,"Creates overlap tables even if empty while loading data","Query execution expects all chunk and overlap tables to exist, even if they are empty. In short term, that means loader should: * look at all chunks and add corresponding overlap chunks, * look at overlap chunks and add missing empty chunk table ",2 DM-2065,"FY17 Implement Data Verification Tool","Need a tool for verifying whether data is in consistent stage (e.g., right after loading, after some upgrades, in general at any given time). The list of things to check include: * empty chunk file, * xrootd exported DB, * data tables * overlap tables, * data_0123456789 tables * chunkId, subChunkId columns existence Some of the above can be automatically fixed on the spot when problem is discovered.",90 DM-2066,"Add test case to catch missing empty chunks or overlaps","Discussed at db hangout 2015-02-18. We need a use case to test for missing empty overlap chunk tables and/or empty chunk tables.",2 DM-2069,"FY17 Design L2 Catalog Swap/Release Automation","Need to think through the issues related to releasing L2 catalog / swapping a new one in place of an old one",54 DM-2070,"FY17 Build AP-ready Data Provenance System","Improvements to the first version of the standalone prototype built through DM-2042. Discussions with the Application Team on capturing provenance and integrating DRP with the provenance system. Add scaffolding / unit tests that will simulate data producers, in particular DRP. Deliverable: DRP-ready System for capturing provenance.",79 DM-2071,"FY18 Integrate AlertProd with Data Provenance","Integrate Alert Production with the Provenance system.",56 DM-2072,"FY17 Implement Async Queries in Data Access Web Services","Work includes: * asynchronous requests, request status, retrieving results for dbserv and imgserv",26 DM-2073,"FY19 Implement Partial Query Results",NULL,100 DM-2075,"S17 Improve ImageServ",NULL,9 DM-2077,"W16 Add Support for Multi-table Shared Scans","Implement multi-table shared scans. Ensure that shared-scans are not delaying interactive queries. The baseline architecture of the shares scans are described in [LDM-135 Shared Scanning|http://ldm-135.readthedocs.org/en/master/#shared-scanning].",100 DM-2078,"F16 Qserv KPMs",NULL,24 DM-2079,"F17 Run Large Scale Qserv Tests",NULL,26 DM-2080,"F18 Run Large Scale Qserv Tests",NULL,26 DM-2081,"FY19 Implement Missing Features in Qserv for L3","Need to think through issues related to supporting Level 3 databases",100 DM-2082,"FY20 Improve Design of Next-to-database Data Analysis","Need to implement the system that will allow users run their own custom data analysis next to database.",100 DM-2083,"FY18 Demonstrate Fault Tolerance",NULL,100 DM-2084,"FY18 Implement Basic Resource Mgmt for DB","Includes things like query throttling per user for all databases (L1, L2, L3)",54 DM-2085,"FY19 Optimize Resource Mgmt for DB",NULL,79 DM-2086,"FY18 Implement Basic Resource Mgmt for Images",NULL,54 DM-2087,"FY19 Optimize Resource Mgmt for Images",NULL,79 DM-2088,"W16 Distributed Loader - Research","In production, we will need a distributed loader that will be capable of loading entire data set produced by DRP within 24-48 hours. This epic involves researching all the needs, requirements and constraints, and exploring what the best architecture for a distributed loader would be. Related doc: [LDM-135 §8.15.2|http://ldm-135.readthedocs.org/en/master/#data-loading]",28 DM-2089,"W16 Distribution and Replica Mgmt Prototype v2","This epic involves building a complete, working prototype of the Qserv data distribution and replica management.",100 DM-2090,"FY18 Implement L2 Catalog Swap/Release Automation","Need to think through the issues related to releasing L2 catalog / swapping a new one in place of an old one",100 DM-2091,"FY17 Add Support for Managing Per-user Access for DB",NULL,53 DM-2092,"FY18 Add Support for Managing Per-user Access for Image and File Archive",NULL,79 DM-2093,"FY18 Integrate Qserv with EFD",NULL,79 DM-2094,"Port metaREST.py to db","metaREST_v0.py in metaserv is currently using MySQLdb instead of going through the db API, because we need to use parameter binding for security reasons. We should switch to using db, once the db interfaces will support it. ",1 DM-2095,"Port dbREST.py to db","dbREST_v0.py in dbserv is currently using MySQLdb instead of going through the db API, because we need to use parameter binding for security reasons. We should switch to using db, once the db interfaces will support it. ",1 DM-2096,"Long term database work planning","Long term planning (updating LDM-240).",8 DM-2097,"Package andyH xssi fixed version (>2MB answer pb) in eups","See DM-1847 - Andy made a patch, it'd be good to the xrootd we use for our stack.",1 DM-2101,"FY18 Revisit L2 Catalog Schema","Revisit the baseline schema",53 DM-2103,"FY18 Implement Internal DRP Database","Implement Internal DRP Database as designed in DM-2038",60 DM-2107,"FY17 Improve ImageServ",NULL,80 DM-2108,"FY19 Demonstrate Qserv Fault Tolerance","Including multi-master failover",100 DM-2110,"FY19 Optimize Partitioning Granularity","We have been always talking about having ~20K chunks per table, and it was driven primarily by spreadsheet-based analysis. We need to look in more details into that, and perhaps even change the model if needed, e.g., introduce different partitioning for larger tables, like ForcedSource.",53 DM-2111,"FY17 Improve Query Coverage in Qserv","Currently Qserv supports only a limited subset of queries. We need to make sure it supports all queries that users need to run.",53 DM-2112,"FY18 Improve Query Coverage in Qserv","Currently Qserv supports only a limited subset of queries. We need to make sure it supports all queries that users need to run.",79 DM-2113,"FY17 Support Explain, Show, List Commands","Implement [explain|http://dev.mysql.com/doc/refman/5.0/en/explain.html] and [show|http://dev.mysql.com/doc/refman/5.0/en/show.html] commands for Qserv. Also, commands such as ""list tables"" will need to be intercepted and overloaded. ",53 DM-2115,"FY19 Implement Multi-master for Qserv",NULL,100 DM-2116,"FY19 Make Database Secure","Revisit security issues, such as sql injections, detecting and shielding from DoS attacks, etc.",79 DM-2117,"FY19 Build/Setup Basic Qserv Monitoring","frontend/worker health monitoring (and management?)",79 DM-2118,"FY18 Implement Failover for L1 Database","Need to implement and test failover - a failure of the master copy of L1 database, and automatic fail over to a replica. The design of the Alert Production L1 database is covered [here|http://ldm-135.readthedocs.org/en/master/#alert-production-and-up-to-date-catalog].",80 DM-2119,"W16 Optimize Secondary Index - Research","Work on the secondary index (objectId --> chunkId / subChunkId mapping). This needs to be scalable to 40B entries. Since we are planning to ingest all data from DRP in <2 days, building should take <2 days. This epic involves researching applicable technologies (including experimenting with most promising ones). Deliverable: proposed technology / architecture along with measures performance at production scale (40 B entries). ",48 DM-2123,"FY18 Add Support for Non-partitioned Tables","Non partitioned tables will need special attention. Options include: a. replicating them on each worker node b. keeping them on a shared file system c. federating Need to thing through these issues, pick the best architecture and implement it.",79 DM-2124,"FY17 Revisit Qserv Deployment on Cluster",NULL,60 DM-2125,"FY18 Design Qserv Software Upgrading","Need to understand how to do software update for Qserv ",26 DM-2129,"FY19 Improve Query Coverage in Qserv",NULL,90 DM-2130,"FY20 Improve Qserv Monitoring","frontend/worker health monitoring (and management?)",79 DM-2131,"Resolve compiler warnings in new measurement framework","When building {{meas_base}}, or any other measurement plugins which follow the same interface, with clang, I see a bunch of warnings along the lines of: {code} In file included from src/ApertureFlux.cc:34: include/lsst/meas/base/ApertureFlux.h:197:18: warning: 'lsst::meas::base::ApertureFluxAlgorithm::measure' hides overloaded virtual function [-Woverloaded-virtual] virtual void measure( ^ include/lsst/meas/base/Algorithm.h:183:18: note: hidden overloaded virtual function 'lsst::meas::base::SimpleAlgorithm::measure' declared here: different number of parameters (4 vs 2) virtual void measure( {code} This is an artefact of a [workaround for SWIG issues|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=20284390]; the warnings aren't indicative of a fundamental problem, but if we can avoid them we should. While we're at it, we should also fix: {code} include/lsst/meas/base/ApertureFlux.h:233:1: warning: 'ApertureFluxResult' defined as a struct here but previously declared as a class [-Wmismatched-tags] struct ApertureFluxResult : public FluxResult { ^ include/lsst/meas/base/ApertureFlux.h:65:1: note: did you mean struct here? class ApertureFluxResult; ^~~~~ struct {code}",1 DM-2136,"W16 Understand Async Queries in Qserv","Understand how disruptive the changes related to implementing asynchronous queries will be for Qserv. Delivarable: brief description outlining changes needed, with story point estimate.",10 DM-2137,"Add parameter binding to db interface",NULL,1 DM-2138,"Validate user query in dbREST","Need to validate query (from security standpoint that user enters through rest api.",1 DM-2139,"Support DDL in MetaServ - implementation","DDL information is embedded as comments in the master version of the schema (in ""cat"" repo). Currently we are only using it for schema browser. This story involves building tools that will load the DDL schema into MetaServ. Design aspects are covered in DM-1770.",8 DM-2141,"Add meas_extensions_shapeHSM to lsstsw, lsst_distrib","meas_extensions_shapeHSM has just been resurrected from bitrot, and should be included in our distribution. Contrary to DM-2140, it should probably not be included in lsst_apps, as it's not clear we want to add a dependency on tmv and GalSim there.",1 DM-2148,"General OpenStack Learning",NULL,6 DM-2149,"Setup spare test hardware for OpenStack testing",NULL,20 DM-2150,"Test Ubuntu OpenStack",NULL,6 DM-2151,"Test Mirantis OpenStack & Fuel",NULL,6 DM-2152,"Figure out OpenStack networking (vLAN, routing, etc)",NULL,6 DM-2153,"Figure out OpenStack integration with LDAP",NULL,17 DM-2155,"Log fails on uniccode string","Log is currently failing if we pass unicode string, it is easy to reproduce by doing: log.info(u""hello""). It fails with: {code} File ""/home/becla/dataArchDev/Linux64/log/master-gfab0203bd3+3/python/lsst/log/log.py"", line 103, in info log("""", INFO, fmt, *args, depth=2) File ""/home/becla/dataArchDev/Linux64/log/master-gfab0203bd3+3/python/lsst/log/log.py"", line 94, in log _getFuncName(depth), frame.f_lineno, fmt % args) File ""/home/becla/dataArchDev/Linux64/log/master-gfab0203bd3+3/python/lsst/log/logLib.py"", line 648, in forcedLog_iface return _logLib.forcedLog_iface(*args) TypeError: in method 'forcedLog_iface', argument 6 of type 'std::string const &' {code}",1 DM-2157,"Data loader crashes on uncompressed data.","Vaikunth just mentioned to me that the is a crash in data loader when it tries to load uncompressed data: {noformat} root - CRITICAL - Exception occured: local variable 'outfile' referenced before assignment Traceback (most recent call last): File ""/home/vaikunth/src/qserv/bin/qserv-data-loader.py"", line 312, in sys.exit(loader.run()) File ""/home/vaikunth/src/qserv/bin/qserv-data-loader.py"", line 248, in run self.loader.load(self.args.database, self.args.table, self.args.schema, self.args.data) File ""/home/vaikunth/src/qserv/lib/python/lsst/qserv/admin/dataLoader.py"", line 168, in load return self._run(d atabase, table, schema, data) File ""/home/vaikunth/src/qserv/lib/python/lsst/qserv/admin/dataLoader.py"", line 192, in _run files = self._gunzip(data) File ""/home/vaikunth/src/qserv/lib/python/lsst/qserv/admin/dataLoader.py"", line 388, in _gunzip result.append(outfile) UnboundLocalError: local variable 'outfile' referenced before assignment {noformat} It looks like we never tested loader on uncompressed data and there is a bug in handling uncompressed data. ",1 DM-2158,"Add support for JSON - define structure","As discussed at [Data Access Hangout 2015-02-23|https://confluence.lsstcorp.org/display/DM/Data+Access+Hangout+2015-02-23], we should support json format. This includes defining the exact format, and implementing it. This story covers defining the format.",2 DM-2159,"Implement Image Response for ImgServ","This story covers implementing proper response, and the header metadata for the fits image response.",3 DM-2161,"Setup webserv for SUI tests","We need to setup a service (eg on lsst-dev) that can be used by the IPAC team to play with our webserv/metaserv/dbserv/imgserv. The server runs on lsst-dev machine, port 5000. To ssh-tunnel, try: {code} ssh -L 5000:localhost:5000 lsst-dev.ncsa.illinois.edu {code} An example usage: {code} curl 'http://localhost:5000/db/v0/query?sql=SHOW+DATABASES+LIKE+""%Stripe%""' curl 'http://localhost:5000/db/v0/query?sql=SHOW+TABLES+IN+DC_W13_Stripe82' curl 'http://localhost:5000/db/v0/query?sql=DESCRIBE+DC_W13_Stripe82.DeepForcedSource' curl 'http://localhost:5000/db/v0/query?sql=DESCRIBE+DC_W13_Stripe82.Science_Ccd_Exposure' curl 'http://localhost:5000/db/v0/query?sql=SELECT+deepForcedSourceId,scienceCcdExposureId+FROM+DC_W13_Stripe82.DeepForcedSource+LIMIT+10' curl 'http://localhost:5000/db/v0/query?sql=SELECT+ra,decl,filterName+FROM+DC_W13_Stripe82.Science_Ccd_Exposure+WHERE+scienceCcdExposureId=125230127' curl 'http://localhost:5000/image/v0/raw/cutout?ra=7.90481567257&dec=-0.299951669961&filter=r&width=30.0&height=45.0' {code} ",2 DM-2163,"Refactor Geom class in Firefly","The Geom class was ported from C code 20 years ago. It needs to refactor to comply with Java OO design. ",8 DM-2164,"Review at DM leadership team meeting","review document with Kantor, KT, Hobblit, and Lambert, including prep time ",3 DM-2165,"Refactor document for that specifications are clearer","1) Have one basic definition of racks and other components in the specifications. 2) Fully write up first full draft specification for the supporting material handing area.",3 DM-2166,"receive / process comments from Jeff Barr a","receive edits from Jeff Barr, accept the formatting and mechanical l edits. Compose separate email list issues related to non LSST tenants in the room. ",1 DM-2167,"Investigate Commerical vendor to deal with comments on requirements. ","process email discussion about the need to liaison with the putative Chilean design contractor. Kantor suggests a contractor to support requirements may be apropos. ",1 DM-2168,"Work inside NCSA to connect procurement contract Modification to OSPRA contract officet","work Jeff's proposal until it reached university contract officer. -- January meeting -- clarify purchasing rules - Internal discussion of property management, - General work within contract modification process. ",6 DM-2171,"Implement JSON Results for MetaServ and DbServ","Implement JSON results for Metadata Service (see all M* in https://confluence.lsstcorp.org/display/DM/API), and Database Service (see all D*) as defined in DM-1868",3 DM-2173,"Disable testDbLocal.py in db if auth file not found","tests/testDbLocal.py can easily fail if required mysql authorization file is not found in user home dir. Skip the test instead of failing in such case.",1 DM-2175,"Adapt integration test to multi-node setup v2","Following DM-595 we can start qserv in multi-node configuration. Next step is to be able to run integration tests in that setup. This needs a bit of understanding how to distribute chunks between all workers in a cluster and how to load data in remote mysql server.",10 DM-2176,"Worker management service - impl","We need to replace direct worker-mysql communication and other administrative channels with a special service which will control all worker communication. Some light-weight service running alongside other worker servers, probably HTTP-based. Data loading, start/stop should be handled by this service.",10 DM-2177,"Implement worker-side squashing","In the port to the new Xrootd Ssi API, worker-side squashing was lost in the shuffle. The plumbing is different, and re-implementing squash functionality is not entirely straightforward, especially because the new API is still missing documentation and examples for implementing cancellation. The consequences of not implementing this are minor--some extra work may be done by the worker, but not a whole lot, because user-level cancellation has not been implemented.",12 DM-2178,"Migrate Qserv to external sphgeom","Migrating qserv to the new c++ geometry API required porting a fair amount of code from the python layer and updating the plumbing in the czar. During implementation, the sphgeom was in the process of finding a home, so the sg code was temporarily placed under core/modules. This ticket covers: * removing core/modules/sg * updating code to point at the external sphgeom * updating build-logic to properly depend on and link with external sphgeom.",4 DM-2181,"S17 Design Prototype EFD Schema for DRP","The epic involves understanding the structure of the EFD database produced by the Engineering and Facility team, and designing schema that will be best suited for Data Release Production. Note that the original EFD database may not even be in MySQL, there were discussions to store it in Postgresql. Deliverable: Alpha version of the EFD database schema for DRP with ""real"" data loaded (if available).",52 DM-2182,"FY18 Design DRP-ready EFD Schema",NULL,79 DM-2186,"Move astrometry_net wrapper code from meas_astrom to meas_astrometry_net","We would like to remove all astrometry.net wrapper code from meas_astrom and put it in a new package with a name such as meas_astrometry_net. This will also require moving any abstract task base classes into a lower-level package such as meas_astrom.",6 DM-2187,"FY17 Data Loader for Large Tables with No Position Information","We need to load some tables (e.g., ForcedSource) that lack director positioning, we will only have the director's primary key. The general case is very expensive (lookup position and chunk for each position), however the fact such tables will be spatially-ordered when loading helps.",60 DM-2188,"Update the astrometry.net astrometry solver to use the new standard schema","DM-1576 provides a new astrometry solver and a new schema for reference objects. However, the old astrometry.net astrometry solver still uses the old schema. It would be wise to convert the old solver to the new schema so that the match list returned by it is in standard format.",4 DM-2189,"Large scale test planning","Need to come up with a plan which data set we will use for large scale tests, and how we will produce it.",10 DM-2190,"Documentation for data loader","Vaikunth had some ""expected"" troubles playing with data loader options for his DM-1570 ticket. Main issue I believe is the absence of the documented use cases and their corresponding data loader options. I'll try to add a bunch of common use cases to RST documentation and also verify that all options behave as expected.",2 DM-2191,"Define command line tasks for pre-ingest transformation","DM-1903 provided a command line task which would transform a {{src}} catalogue into calibrated form. Here, we build on that to provide command line tasks for all source catalogues which will need to be ingested; will include at least {{deepCoadd_src}}, {{goodSeeingCoadd_src}}, {{chiSquaredCoadd_src}}.",6 DM-2192,"Provide transformations for ""big three"" measurements","Provide standard calibration transformations for each of shape, flux and centroid and make sure they are returned as the default transformation for all algorithms measuring those quantities.",10 DM-2193,"Add assertXNearlyEqual to afw","We often want to compare two WCS for approximate equality. afw/image/testUtils has similar functions to compare images and masks and I would like to add one for WCS This ended up being expanded to adding functions for many afw classes (not yet including image-like classes, though existing functions in image/testUtils for that purpose should probably be wrapped or rewritten on a different ticket)",5 DM-2194,"Ensure proper functioning of HSC distortion correction within obs_subaru","There may be some discrepancy between the pixel units being passed to distest.cc compared to what it is expecting (units of pixels). This needs to be investigated further and remedied in such a way that all other representations (e.g. in camera.py) are consistent with the other obs_XXX representations.",6 DM-2195,"Create form framework in React","We want to create a new frame work for entering data for forms and dialogs. This is in javascript based on React.js. This is the first step in our javascript conversion.",10 DM-2197,"Prototype HTM-based spatial binning to visualize large number of catalog sources","See story DM-1551.",8 DM-2199,"Build 2015_03 Qserv release","See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe. ",1 DM-2201,"Add typemaps for numpy scalars","Add typemaps so that we can use numpy scalars to call C++ functions that take plain old scalar types (e.g. float, double or int). At present attempting to pass numpy scalars will fail unless the type is one of a restricted subset, e.g. float and numpy.float64 succeed but numpy.float32 is rejected as being an incompatible type, and similarly for integer types.",4 DM-2202,"Acquire development data","We'll need a reference set of data to work against. This could be SDSS, CFHT, or simulated. Should be 10? epochs with realistic atmospheric conditions taken at similar airmass and hour angle. Single band is fine for now.",4 DM-2203,"Produce task API","This will require a new task, so will require a new interface and associated RFCs. The interface should take an arbitrarily large stack of catalogs with or without a reference catalog. It should return a stack ow WCSs that map from the individual coordinate systems to the reference.",6 DM-2205,"Break down monster DM-1108 stories","[~pgee] -- After finishing the measurement work, your next priority is to get started on DM-1108. However, the stories you have been assigned there are currently too big for useful scheduling (20-30 SPs is a mini-epic; we're looking for less than 10 SPs per story). The first task therefore is to work with [~jbosch], and others if required, to break them down and come up with a set of stories which usefully reflect the work which needs to be done.",4 DM-2206,"Deploy and test network emulation for nightly processing testbed","Deploy and test network emulation for nightly processing testbed. Assignees: Paul Wefel, Steve Pietrowicz, James Parsons Duration: January - February 2016",19 DM-2207,"Alert Production Simulator","Start March 2015, finish July 2015 Pietrowicz S - 100%",96 DM-2208,"Complex Event Processing","Start May 2015, finish June 2015",31 DM-2209,"OCS Software","Start July 2015, finish August 2015 Pietrowicz S - 100%",10 DM-2210,"Configuration Management (Puppet)","Start March 2015, finish May 2015 Mather B - 40%",9 DM-2211,"Setup qserv prototype for qserv & SUI teams","Start July 2015, finish August 2015 Glick B - 25% Qserv requirements: - SUI will be testing against lsst10 (or IPAC qserv) for now ? SUI requirements: Xiuqin's 'short term' version: - 1 VM - SUI build server - 4GB memory and 200GB hard disk should be good enough. - 1 VM - Apache server as a proxy and web front end - 4GB memory and 100GB hard disk should be enough - port 80 accessible from outside - 2 VMs - Tomcat servers - each has 16GB memory, access to 1TB of shared hard disk - Port 8080 should be open for Apache server to access - Port 8009 should be open to each other so they can replicate cache. (First 2 VMs are not absolutely needed. We can always use one of the hosts in number 3 to do build and host Apache server.) Trey Roby's 'long term' (in 2+ years) SUI requirements: - 2 vm/machines for Tomcat servers, they are fairly large - each 100 GB mem - each 16 processors - 1 TB disk space shared and accessible between. - 1 vm/machine for Web Server, can be small - 16 GB mem - 30 GB disk - 4 processors - 2 small vm/machines for playing around with workspaces/L3 concepts - each has 8 GB mem - each 10 GB disk - 4 processors - Can share the Tomcat servers disk space",10 DM-2213,"Storage Policies and Alignment","Start March 2015, finish August 2015 Freemon M - 100%",50 DM-2214,"File System Research and Prototyping","Start March 2015, finish September 2015 Freemon M - 100%, Glick B - 25%, Daues G - 40%, Elliot M - 25%",100 DM-2215,"File Management Technology","Start March 2015, finish September 2015 Daues G - 40%, Freemon M - 100%",50 DM-2216,"Understand GPFS and commercial filesystems between data centers","Start May 2015, finish June 2015 Petravick D - 50%, TBD from SET group",6 DM-2217,"Update Sizing Model","Start March 2015, finish September 2015 Alt J - 50%, Petravick D - 10%",22 DM-2218,"Base Data Center Requirements","Start March 2015, finish May 2015 Petravick D - 50%",18 DM-2221,"Start understanding inheritability and reusability of dataset types","In order to allow for on-the-fly Task creation of dataset types, the essentials of each type need to be encapsulated in code. That code should be reused across all similar dataset types, and there are opportunities for inheritance and specialization, particularly in cases like simple file-oriented mappers. Investigate this by prototyping a number of possibilities.",4 DM-2224,"Wide-Area Network Work","Start March 2015, finish September 2015 Wefel P - 25%",20 DM-2225,"LOE - S15 (sys admin)","Glick B - 25%, Mather B - 40%, Elliot M - 25%, Freemon M - 100% Wefel P - 25%",80 DM-2226,"LOE - S15 (management)","Petravick D - 50%, Gelman M - 50%",20 DM-2227,"LOE - S15 (misc)","All NCSA team",10 DM-2228,"LOE - Week ending 3/6/15","- backup issues with lsst-stor141 (https://jira.ncsa.illinois.edu/browse/LSST-632) - setup jumbo frames on lsst-xfer (https://jira.ncsa.illinois.edu/browse/LSST-628) - crashplan reconfig on lsst-xfer (https://jira.ncsa.illinois.edu/browse/LSST-629)",8 DM-2229,"LOE - Week ending 3/13/15","- crashplan issue with lsst-netem (https://jira.ncsa.illinois.edu/browse/LSST-633) - yum/glibc issue with lsst-dbdev1 (https://jira.ncsa.illinois.edu/browse/LSST-631) - account for Jacques Sebag (https://jira.ncsa.illinois.edu/browse/LSST-624)",8 DM-2230,"LOE - Week ending 3/20/15","- lsst-dbdev2 drive failure (https://jira.ncsa.illinois.edu/browse/LSST-636) - account for Colin Slater (https://jira.ncsa.illinois.edu/browse/LSST-634) - disable Robyn Allsman's accounts (https://jira.ncsa.illinois.edu/browse/LSST-623)",26 DM-2231,"LOE - Week ending 3/27/15",NULL,21 DM-2232,"LOE - Week ending 4/3/15","- researched buildbot slowness on lsst-dev - researched buildbot slowness on lsst-dev ",16 DM-2233,"LOE - Week ending 4/10/15",NULL,16 DM-2234,"LOE - Week ending 4/17/15",NULL,17 DM-2235,"LOE - Week ending 4/24/15","- Researched how to monitor network drops, errors, etc - Opened up SUI ports on lsst-dev - Moved /nfs/admin/ to /condo/admin/ - Review of EL 6.x kernel security patch - Fixed jumbo frame issues, primarily with old VMs that needed new version of NIC ",15 DM-2236,"Prototyping with Puppet",NULL,5 DM-2237,"Test Puppet with base configuration manifests",NULL,38 DM-2239,"Develop use cases for TOWG","Start March 2015, finish May 2015 Petravick D - 50%, Glick B - 25%, Gruendl R - 5%",20 DM-2240,"ISO Work","Start March 2015, finish September 2015 Withers A - 25%",40 DM-2243,"Extend API: expose cursor","Extend API to expose cursor. This was brought up by Andy in DM-2137. ",1 DM-2245,"Define ntermediate plan for MacOSX builds"," We have 1. Obtain a dedicated colo OSX server 2. Have done some testing using the SQuaRE vagrant-sandbox harness It is therefore a plausible avenue forward to do at least a nightly build/deploy/intgeration-test on OSX pending more extensive arrangements requiring purchase of hardware. ",4 DM-2246,"Github transition for DM","DM's transition for code repositories to Github is complete. Outstanding are data repositories; a cleanup of contrib/externals; and supporting the Stash move. ",10 DM-2247,"Workflow improvements for Sims / PST projects","New wokflow for Sims Merge of Opsim and CATsim New workflow for PST ",5 DM-2248,"Prototype automated system for release preparation builds"," Prototype an environment that allows automatic - Provisioning of a VM for a certain OS - Install the Stack prerequisites for that OS - Build the stack via newinstall.sh from the production server - Run integration tests (in the curent case the sdss test https://github.com/lsst-sqre/sandbox-stackbuild",22 DM-2250,"Galaxy Fitting via ""ngmix""","Provide wrappers that let us run Erin Sheldon's [ngmix|https://github.com/esheldon/ngmix] as part of the DM pipeline. Issues so far only cover getting the a single-frame (visit or coadd) version of the code running. ngmix can also to simultaneous fitting to multiple exposures, but it's not yet clear how we'll want to handle the I/O and that interacts with a future multifit plugin framework.",45 DM-2251,"Implement SExtractor's SPREAD_MODEL","The new SExtractor star/galaxy classifier, SPREAD_MODEL, is popular with everyone who has tried it, and should be simple to implement by building on code in meas_modelfit. See definition and discussion here: http://arxiv.org/abs/1306.4446",2 DM-2252,"Define common interface for star/galaxy classifiers","We need some common fields for star/galaxy classifiers so they can participate in a slots-like mechanism once we have several of them. Most of these can produce a floating point number between 0 and 1 (but sometimes it's not limited to that range), and it's rarely a true probability. We may want to make a boolean that results from a threshold on these be the common interface, but we don't necessarily want to hard-code such a threshold into the processing either - especially when we could also use a FunctorKey to get a boolean from the floating point value.",2 DM-2253,"add third-party package builds for ngmix dependencies","In addition to numpy and scipy, ngmix depends on the emcee and statsmodel packages. While it can build without them, we probably want the full functionality. I also see some undeclared dependencies on the ""esutil"" and ""fitsio"" packages (all from esheldon's GitHub), and there may be a few more dependencies on some of his own packages. This issue includes creating a third-party build for ngmix itself.",6 DM-2254,"Add SFM plugin for ngmix MCMC sampling","Add an SFM plugin for ngmix MCMC fitting, as in the example in the ngmix README. This should depend on DM-5429 (or a suitably configured modelft_ShapeletPsfApprox) for PSF approximation. For now, we should just take the mean of all parameters in the MCMC samples and write those to the record, as we currently don't have any way to save all of the samples. Testing and tuning this algorithm to get it working well should be deferred to another issue. The only requirement here is that it be able to run without crashing (even if that means setting the number of samples small).",10 DM-2256,"make a simple build for Firefly package","We want to have a out of box build for users of Firefly package. It will include a simple Firefly viewer. ",6 DM-2257,"Allow eups xrootd install script to be relocatable","xrootd lib/ directory should be s relative symlink to lib64, no a full path link.",1 DM-2258,"Setup in2p3 cluster for Qserv team","- create accounts - update umask on stack to each account - provide easy ssh config if possible - setup up build procedure (each developer can build Qserv using tag git and 'git' version is set up by default on all the Qserv if ti exists)",2 DM-2259,"remove PSFAttributes","PSFAttributes has long been deprecated, and we just need a little more work to remove it: - Add an effective area accessor to the Psf interface, and implement it in ImagePsf. - Replace usage of PSFAttributes with usage of Psf accessors. This may require a little work if code depends on the details of how the shape was calculated, as PSFAttributes provided support for more algorithms than we will going forward.",2 DM-2262,"Improve build system for sphgeom",NULL,2 DM-2263,"Create pilot condor jobs","Create long-running jobs to reduce the startup time for new HTCondor jobs. This can be implemented as a parent/child, or as a on_exit_remove=false directive in HTCondor. I suspect it will be a combination of the two.",21 DM-2264,"Implement task switching between work job machines","AP requires that jobs are handed off to different worker job clusters as the previous set of images is being worked on.",4 DM-2265,Refactoring,"The initial prototype of the AP simulator needs to be refactored to improve how tasks are handled by the components for further development.",25 DM-2268,"Implement API for reading simulated camera data","Currently this is generated by the replicator and sent to the distributor. The idea where is to put the API in place so that the data will be transferred from outside of the replicator to it, and then passed on.",10 DM-2269,"Implement file transfer API ","Create file transfer API so we can easily test different types of file transfer mechanisms to/from the AP.",4 DM-2270,"Move VMs to Docker containers","We anticipate being able to move from the VMs that we currently use to using docker. This will require some coordination with Greg Daues to see how HTCondor is configured. ",2 DM-2272,"Unify logging strategy for python scripts","- add -vvv option - remove default value for configuration file in logger, provide it at each script level (i.e. integration test, data loader). - if it exists, provide configuration file option explicitly to all called submodules which uses it. See admin/python/lsst/qserv/admin/logger.py {code:python} 14 def get_default_log_conf(): 15 default_log_conf = ""{0}/.lsst/logging.ini"".format(os.path.expanduser('~')) 16 return default_log_conf 17 18 def add_logfile_opt(parser): 19 """""" 20 Add option to command line interface in order to set path to standar 21 configuration file for python logger 22 """""" 23 24 parser.add_argument(""-V"", ""--log-cfg"", dest=""log_conf"", 25 default=get_default_log_conf(), 26 help=""Absolute path to yaml file containing python"" + 27 ""logger standard configuration file"") 28 return parser 29 30 31 def setup_logging(path='logging.ini', 32 default_level=logging.INFO): 33 """""" 34 Setup logging configuration from yaml file 35 if the yaml file doesn't exists: 36 - return false 37 - configure logging to default_level 38 """""" 39 if os.path.exists(path): 40 with open(path, 'r') as f: 41 logging.config.fileConfig(f) 42 return True 43 else: 44 logging.basicConfig(level=default_level) 45 return False {code}",6 DM-2277,"Document HOW-TO setup-up krb5 for easy cluster access","{code:bash} su aptitude install krb5-user # edit /etc/krb5.conf w.r.t ccage one # then as desktop user kinit ssh ccqservxxx {code} /etc/krb5.conf {code:bash} [libdefaults] default_realm = IN2P3.FR ... allow_weak_crypto = true ... [realms] IN2P3.FR = { kdc = kerberos-1.in2p3.fr:88 kdc = kerberos-2.in2p3.fr:88 kdc = kerberos-3.in2p3.fr:88 master_kdc = kerberos-admin.in2p3.fr:88 admin_server = kerberos-admin.in2p3.fr kpasswd_server = kerberos-admin.in2p3.fr default_domain = in2p3.fr {code} sshconfig: {code:bash} Host ccqservbuild GSSAPIAuthentication yes GSSAPIDelegateCredentials yes ForwardX11 yes HostName ccqservbuild.in2p3.fr #ProxyCommand ssh -W %h:%p cc Host ccqserv1* GSSAPIAuthentication yes GSSAPIDelegateCredentials yes ForwardX11 yes HostName %h.in2p3.fr ProxyCommand ssh -W %h:%p ccqservbuild {code}",2 DM-2279,"Fix problems with mysql timeout","We added some code for supporting reconnecting (see https://dev.lsstcorp.org/trac/ticket/3042) but clearly not enough to recover from connection timeouts. This needs to be addressed.",1 DM-2280,"The TAN_PIXELS cameraGeom coordinate system should be with respect to the center of the focal plane","The TAN_PIXELS cameraGeom coordinate system (the position on a detector if there is no optical distortion) is presently defined with respect to the center of the detector -- i.e. a star at the center of the detector will have the same position in PIXELS and TAN_PIXELS coordinates. That is a mistake. TAN_PIXELS should be defined with respect to the center of the focal plane, since it then reflects the effects of having optical distortion or not. Fixing this will help meas_astrom match stars. The effects of not fixing it are making the matcher search farther for a fit. As long as we allow sufficient offset in the matcher config the current system will work, but it is not ideal.",2 DM-2281,"Implement connection pool","Implement a class that manages a connection pool, and optionally, if configured, restarts connection as needed in case of timeout.",1 DM-2282,"Switch to using db connection pool","Switch to using the db connection pool. Note, in addition to getting auto-reconnect, in metaserv that would handy if we need to talk to multiple database servers simultaneously.",1 DM-2286,"Participate in design process","Participate and guide the SUI design process, generate charts and documents as appropriate",9 DM-2287,"Move javascript code into firefly repo and begin creating a real input form",NULL,10 DM-2288,"Work with Camera & Pipeline team to spec out proof of concept tools",NULL,14 DM-2289,"Personnell requisitions ","Work though recruiting for software effort. Investigated and filled the ""kenton"" recruiting pattern at NCSA -- (few explicit requirements, many desirable) Began discussion to break down hires for ""2nd"" floor work -- to be in the LSST group v.s support groups -- ADS and Doug's group",3 DM-2290,"Arrange for commercial object store presentation","arrage for presentations next week w.r.t commercial object store. The vendor in question is know to NCSA and has claims to have produced a commercial object store having both NFS, GPFS and swift interfaces. ",1 DM-2291,"Begin WBS review ","Begin comprehensive review of the WBS. Forced on overall framework and begin workflow systems ",1 DM-2292,"Security officer orientation","begin orientation of LSST ISO ALEX Withers. ",1 DM-2293,"Internet2 TIER investigation",NULL,1 DM-2294,"Unable to start cmsd on Qserv worker node","Some build issues have qlready been fixed in commit: 9dd378829e8751a6852356967411c20580e2a1c3 Here's the log: {code:bash} [fjammes@ccqserv101 ~]$ cat /qserv/qserv-run/var/log/worker/cmsd.log 150309 21:19:46 9794 Starting on Linux 3.10.0-123.8.1.el7.x86_64 Copr. 2004-2012 Stanford University, xrd version v20140617-203cf45 ++++++ cmsd worker@ccqserv101.in2p3.fr initialization started. Config using configuration file /qserv/qserv-run/etc/lsp.cf =====> all.adminpath /qserv/qserv-run/tmp =====> xrd.port 1094 =====> xrd.network nodnr Config maximum number of connections restricted to 4096 Config maximum number of threads restricted to 2048 Copr. 2007 Stanford University/SLAC cmsd. ++++++ worker@ccqserv101.in2p3.fr phase 1 initialization started. =====> all.role server =====> ofs.osslib libxrdoss.so =====> oss.localroot /qserv/qserv-run/xrootd-run =====> cms.space linger 0 recalc 15 min 10m 11m =====> all.pidpath /qserv/qserv-run/var/run =====> all.adminpath /qserv/qserv-run/tmp =====> all.manager ccqserv100.in2p3.fr:2131 =====> all.export / nolock The following paths are available to the redirector: w / ------ worker@ccqserv101.in2p3.fr phase 1 server initialization completed. ++++++ worker@ccqserv101.in2p3.fr phase 2 server initialization started. Plugin Unable to find required version information for XrdOssGetStorageSystem in osslib libxrdoss.so ------ worker@ccqserv101.in2p3.fr phase 2 server initialization failed. 150309 21:19:46 9794 XrdProtocol: Protocol cmsd could not be loaded ------ cmsd worker@ccqserv101.in2p3.fr:1094 initialization failed {code}",2 DM-2295,"Read through Don's SCADA notes and comment",NULL,1 DM-2299,"Revisit db and dbPool, separate connection from utilities","Revisit whether we need something better than a very basic db connection pool. It may be worth looking at http://docs.sqlalchemy.org/en/rel_0_9/core/pooling.html (or even sqlalchemy in general). Note the Pooling Plain DB-API Connections section - one can use sqlalchemy pooling independently of the other library features. Separate utilities like createDb(), dbExists() and such from core part that deals with connections / sqalchemy.",12 DM-2300,"Improve Webserv/Metaserv","Work includes implementing features requested by SUI (schema metadata, units etc)",35 DM-2301,"Support metadata for databases without description","The metaserv should be able to support databases for which we don't have the ascii schema with descriptions and special tokens (ucd, units etc). This story involves implementing it. In practice, the metaserv/bin/metaBackend will need to be extended to implement ""ADD DB""",4 DM-2305,"Measurement transforms for Flux","Provide calibration transforms for flux measurements to magnitudes.",3 DM-2306,"Measurement transforms for centroids","Provide calibration transforms for all algorithms measuring centroids.",5 DM-2307,"Measurement transforms for shapes","Provide calibration transforms for algorithms measuring shapes.",2 DM-2309,"Update dev quick-start guide to new git repositories","The quick-start documentation for developers still points to the old git repositories. The RST document needs to be updated to the GitHub repos.",1 DM-2312,"obs_test's table file is out of date","obs_test's table file is somewhat out of date. Problems include: - afw is required but missing - meas_algorithms and skypix are used by bin/genInputRegistry.py, which is only used to create the input repo so these can be optional - daf_persistence is not used - daf_base is only used by bin/genInputRegistry.py, so it can be optional (though it is presumably setup by daf_butlerUtils in any case)",1 DM-2314,"Improve xssi API to send a few bytes with the message informing the client that a response is available on the server","This would allow Qserv no to send the first protobuf header as a xrootd in-band message, and save some resources (network and CPU due to xrootd/TCP/IP encapsulation)",6 DM-2316,"Clarify expectations for unauthenticated user data access","h4. Short version: Clarify what existing community practices, notably including VO interfaces, appear to rely on the availability of unauthenticated access to information in astronomical archives. h4. Details: At the February DM All Hands, [~frossie] raised an objection when it was mentioned that there is a presumption that all user access to LSST data through the DM interfaces (as opposed to through EPO) will be authenticated. We don't appear to have ever documented an explicit requirement that all access be authenticated. The basic controlling requirement is OSS-REQ-0176, ""The LSST Data Management System shall provide open access to all LSST Level 1 and Level 2 Data Products, as defined in the LSST System Requirements and herein, in accordance with LSSTC Board approved policies. ..."", which was a carefully crafted indirection at a time when the policy for non-US/Chile access was still being developed. However, this presumption has been around for a long time. It is inherent to the project policy that access to the non-Alert data will be limited to individuals who are entitled to it. No matter what we think the final policy might be, we do have to design a system that can be consistent with this policy. [~frossie] stated that the astronomical community relies on certain types of data and metadata - she mentioned coverage maps, among others - being available through unauthenticated interfaces. This ticket is to ask her (and others) to collect documentation of those existing practices, so that we can figure out what the expectations may be and how to respond to them in our design.",2 DM-2320,"Remove deprecated merging code: rproc::TableMerger","rproc::TableMerger seems to be replaced with rproc::InfileMerger, so this class could certainly be removed easily. ",2 DM-2322,"Revisit exceptions in db module","Revisit db/python/lsst/db/exception.py. Perhaps get rid of the numbers.",5 DM-2323,"KT reading list for operational requirements",NULL,2 DM-2324,"Observatory site requirements reading",NULL,2 DM-2327,"Setup hosts for SUI (2x Tomcat, Apache, and build)","Xiuqin's 'short term' version: 1 VM - SUI build server 4GB memory and 200GB hard disk should be good enough. 1 VM - Apache server as a proxy and web front end 4GB memory and 100GB hard disk should be enough port 80 accessible from outside 2 VMs - Tomcat servers each has 16GB memory, access to 1TB of shared hard disk Port 8080 should be open for Apache server to access Port 8009 should be open to each other so they can replicate cache. (First 2 VMs are not absolutely needed. We can always use one of the hosts in number 3 to do build and host Apache server.) The 2 Tomcat servers are larger than we can currently support as VMs. We've discussed repurposing 2 of the older LSST ""cluster/condor"" nodes (e.g. lsst14 & lsst15) for this purpose. But, ideally these could be implemented with the new vSphere hardware if the timeframe works.",4 DM-2329,"review """"data center in a box"" mali. Recover consultant's contact into ","reviewed the data center in a box, recovered consultant's name prior to drafting a SOW.",1 DM-2330," attend DDN WOS briefing, write summary note. ","as described above. Summary note is attached. also looked for use of this product in DOE labs, who would be consumers of LSST data. Discovered that it had been investigated for use in HEP a few years earlier, but that is was not adopted because, at that time the hardware and software were coupled.",1 DM-2331,"misc for week of march 9","finalize job descriptions. Meet with kantor additional hour of orientation for the ISO. group meeting Misc.",1 DM-2334,"Simplify interactions with XrdOss","The qserv code is still using the old ssi scheme for the cmsd, this needs to be rewritten. For details, see https://listserv.slac.stanford.edu/cgi-bin/wa?A1=ind1503&L=QSERV-L#3",5 DM-2335,"Setup IRODS zone on ISL OpenStack","We begin an examination of iRODS for managing data collections. We perform initial testing using resources available on NCSA's ISL OpenStack. To mock up a zone or 'data grid' managed by iRODS, we set up an ICAT server, a resource server (this is a data storage resource that does not run the central database), and a client host. ",4 DM-2336,"Save iRODS installations/servers as docker images","We install and configure iRODS servers (an ICAT server, a resource server, a client host) in docker and make images, pushing the results to a docker hub repository. ",4 DM-2337,"develop/propose storage policies",NULL,6 DM-2338,"develop/propose storage procedures",NULL,10 DM-2339,"develop/propose storage implementation",NULL,3 DM-2340,"Reprise SDRP processing metrics","In support of an SDRP-based science talk of Yusra AlSayyad, we spent some cycles gathering/summarizing processing middleware results and metrics from the US side of processing of the Split DRP. This information from notes, logs, databases, etc provided contextual information on the processing campaign that produced the SDRP science results. ",2 DM-2341,"Use parallel ssh to manage Qserv on IN2P3 cluster","IN2P3 sysadmin won't manage Qserv through puppet. So Qserv team has to provide ssh scripts to do this. ",5 DM-2342,"Integrate changes from Events code review",NULL,6 DM-2343,"Move afw_extensions_rgb functionality into afw proper","See RFC-32 ",1 DM-2347,"(In)equality semantics of Coords are confusing","Viz: {code} In [1]: from lsst.afw.coord import Coord In [2]: c1 = Coord(""11:11:11"", ""22:22:22"") In [3]: c1 == c1, c1 != c1 Out[3]: (True, False) In [4]: c2 = Coord(""33:33:33"", ""44:44:44"") In [5]: c1 == c2, c1 != c2 Out[5]: (False, True) In [6]: c3 = Coord(""11:11:11"", ""22:22:22"") In [7]: c1 == c3, c1 != c3 Out[7]: (True, True) {code} {{c1}} is simultaneously equal to *and* not equal to {{c3}}!",1 DM-2348,"useValueEquality and usePointerEquality fail to fail","These SWIG macros return a class instead of raising an exception instance when the equality operation fails.",1 DM-2349,"Add unit tests to SchemaToMeta","Add unit tests, also improve variable names as suggested by K-T in comments in DM-2139",1 DM-2352,"Install and learn to use iPython notebook",NULL,4 DM-2353,"Participate in design discussion","Participate in the design discussions three times weekly for two to three months. ",9 DM-2354,"Participate in design discussion","Participate in the design discussions three times weekly for two to three months. ",9 DM-2355,"Participate in design discussion","Participate in the design discussions three times weekly for two to three months. ",9 DM-2356,"Identify the hardware resources needed at NCSA for short term development ","Supply the hardware resources needed at NCSA for short term development. It is captured in DM-2327 ",1 DM-2357,"make PixelFlagsAlgorithm fully configurable","PixelFlagsAlgorithm currently hard-codes the mask planes it considers. This should be fully configurable instead. It also overloads the ""edge"" flag to mean both ""EDGE mask plane was set"" and ""centroid was off the edge of the image"". These should be different flags. We may also want to have this algorithm use SafeCentroidExtractor.' Finally, the algorithm is woefully undertested.",2 DM-2358,"standardize handling of missing peaks in centroiders","GaussianCentroid has a NO_PEAK flag that it sets when there is no Peak to use as an input. SdssCentroid does not. This behavior should be standardized. Maybe we should use SafeCentroidExtractor here?",1 DM-2363,"RGB code introduces dependency on matplotlib","While the new RGB code looks like it's just calling NumPy, NumPy is actually delegating to matplotlib under the hood when it writes RGB(A) arrays. It also turns out that code is broken in matplotlib prior to 1.3.1 (though that shouldn't be a problem for anyone but those who - like me - are trying to use slightly older system Python packages). I think think this means we should add an optional dependency on matplotlib to the afw table file, and condition the running of the test code on matplotlib's presence (and, ideally, having the right version). I'm happy to do this myself (since I'm probably the only one affected by it right now).",1 DM-2364,"Revisit the choice of using flask","We should quickly revisit if flask is the right choice for us. Related: reportedly, our simple flask-based webserver is using more CPU in an idle state than expected. It might be useful to profile things, and look into that. ",1 DM-2367,"run lsstswBuild.sh in a clean sandbox","The ""driver"" script, lsstswBuild.sh, used by the buildbot slave on lsst-dev to initiate a ""CI run"" has a number of environment assumptions (binaries in the $PATH, paths to various components, hostnames, etc.). This prevents it from [easily] being invoked on any other host. As lsstswBuild.sh builds a number of packages that are not in the lsst_distrib product, the os level dependencies for these other products need to be determined. In addition, the current version of lsstswBuild.sh and related scripts on lsst-dev are not version controlled.",8 DM-2370,"Move QuerySession::_stmtParallel from query::SelectStmtPtrVector to query::SelectStmtPtr","QuerySession::_stmtParallel is a vector but it seems only it's first element is used, so storing it in a vector doesn't seem necessary. Code can be easily simplified here. This should lead to mode understandable code. QuerySession public members and method comments could also be improved here.",4 DM-2371,"run lsstswBuild.sh under Jenkins on EL6","* Demonstrate lsstswBuild.sh being invoked by jenkins on EL6 (same OS as lsst-dev). * Experiment with a single build slave attached to a jenkins master * Investigate configuration management of jenkins builds.",6 DM-2373,"Improve logger use in qserv","Qserv logger must be easily configurable. Next technique, based on log4cxx documentation allows to do it easily. Example: In QuerySession.cc, initialize a static logger: {code:c++} namespace lsst { namespace qserv { namespace qproc { LOG_LOGGER QuerySession::_logger = LOG_GET(""lsst.qserv.qproc.QuerySession""); {code} then use it in Query session member functions: {code:c++} if (LOG_CHECK_LVL(_logger, LOG_LVL_DEBUG)) { std::ostringstream stream; _showFinal(stream); LOGF(_logger, LOG_LVL_DEBUG, ""Query Plugins applied:\n %1%"" % stream.str()); } {code} And use log4cxx.property to easily configure, AT RUNTIME, logging for each Qserv module class: {code} # logger for all module will inherit this one log4j.logger.lsst.qserv=ERROR # this could be generalized to all Qserv modules: log4j.logger.lsst.qserv.qproc=INFO # can also be done at the class level for advanced debugging #log4j.logger.lsst.qserv.qproc.QuerySession=DEBUG {code} And then in the log: {code} /home/qserv/qserv-run/2015_03/var/log/qserv-czar.log:0319 17:08:48.786 [0x7f208da93740] DEBUG lsst.qserv.qproc.QuerySession (build/qproc/QuerySession.cc:118) - Query Plugins applied: {code} This proposal is a draft and should be improved before implementing it.",8 DM-2375,"evaluate NCSA proposal to investigate CEPH in the context of NCSA Integrated systems lab","The integrated systems lab (ISL) is the orgianizational vehicle used to investigate pre-production technologies at NCSA. Since We still lack the ability to procure goods, I evaluated and commented on an ISL proposal to investigate the CEPH file system for its properties as an alternative to the LSST baseline file system GPFS. ",1 DM-2376,"revise and circulate data center requirements note","Reconvert the ~10 TBD's in the priori version of the note — I’ve kept the stipulation that end of service-lifee stuff will leave these spaces but added an appendix that “this is what the central space should provide” My understanding is there are now discussions on whether that central space will exist of not. The requirements can be promoted to center requirements no central space is evient. — The maximal average weight for a rack was computed from LDM-144 and is given. — There are more cu ft estimates for the need to dispose of dunnage and packing material. — TBD’s w.r.t overhead cabling are specificed. — The non- LSST requrements are in there, and have been as far as I am able to ascertain them from champaign urbana. Ron has been stating requirements as “rows” I have never fixed row length thinking that is for the designer to do. We’ve stated that rows are shareable, but racks are not. so what’s in the docs is definitive unless/until non LSST requirements can be stated in the same terms. ""Shall support 16 racks for the NOAO tenant"". is what I had. power requirements as per the common space, becasue we want a maximally flexible space. ",2 DM-2377,Management,"Meetings -- Monday CAM meeting, Friday standup and infrastructure. Internal NCSA group meeting, Internal NCSA ""comp pol"" technical coordination meeting. Screen existing candidate pool for likely people to fill opening, Interviewed one person checked references + Misc.",3 DM-2380,"Retrieve HSC engineering data","HSC data becomes public 18 months after it was taken, so data taken during commissioning are now available. We would like to use this data for testing the LSST pipeline. It needs to be downloaded from Japan.",2 DM-2382,"Make sure the command-line parser warns loudly enough if no data found","A user recently got confused when calling parseAndRun didn't call the task's run method. It turns out there was no data matching the specified data ID. Make sure this generates a loud and clear warning.",1 DM-2383,"migrate package deps from sandbox-stackbuild to a proper puppet module","There is a growing list of known package dependencies in the sandbox-stackbuild repo and a need to use this information for independent environments (such as CI). This list of packages should be lifted out into an independent puppet module that can be reused.",2 DM-2385,"Implement data loading in worker manager service","This is a separate ticket for implementation of the data loading part of the worker management service (started in DM-2176). Some ideas and thoughts are outlined in that ticket.",6 DM-2387,"Build testQDisp.cc on ubuntu","testQDisp.cc needs flags -lpthread -lboost_regex to build on ubuntu.",1 DM-2390,"Errors need to be checked in UserQueryFactory from QuerySession objects","UserQueryFactory doesn't check its QuerySession object for errors after setQuery. Thus it continues setting things up after the QuerySession knows the state is invalid.",1 DM-2391,"Migrating to GWT 2.7","To use JavaScript Interop functionality, we need to migrate to GWT 2.7",10 DM-2392,"React components for Form","To familiarize myself with React and to prepare the ground for moving to React-based user interfaces, we need to create React components for the form. This story includes CheckboxGroup, RadioGroup, and Listbox input fields.",10 DM-2396,"FY17 Enable DC Analyses through Qserv","Load data challenge data into Qserv and enable analytics of the DC data through Qserv.",53 DM-2397,"FY18 Enable DC Analysis through Qserv","Load data challenge data into Qserv and enable analytics of the DC data through Qserv. ",53 DM-2400,"FY17 Fix Qserv Bugs","Bucket epic for unexpected bug fixes.",53 DM-2401,"FY18 Fix Qserv Bugs","Bucket epic for unexpected bug fixes.",53 DM-2402,"FY19 Fix Qserv Bugs","Bucket epic for unexpected bug fixes.",53 DM-2403,"FY20 Fix Qserv Bugs","Bucket epic for unexpected bug fixes.",100 DM-2404,"W16 Butler (v4)",NULL,100 DM-2411,"Allow qserv-admin.py to delete a node","Registered workers in CSS with qserv-admin.py are currently not able to be removed (no DELETE NODE type command). Also, changing node status from ACTIVE to INACTIVE needs to be fixed.",1 DM-2412,"Change integration test user from root to qsmaster","Currently integration tests use root account as default user - this should be changed to qsmaster for the future.",2 DM-2414,"investigate configuration management for jenkins","The most popular Puppet module for managing Jenkin's {code}jenkinsci/puppet-jenkins{code} is able to create a master and build slaves but is missing the functionality to manage several master configuration options that otherwise require manual setup. We need to investigate the difficulty of managing a Jenkin's master configuration values in an idempotent manner.",4 DM-2415,"convert Statistics to use ndarray natively","The Statistics class predates ndarray, and hence uses some hackish Image-class emulators/wrappers to deal with 1-d arrays. It'd clean things up considerably to have it use ndarray under the hood, and have the Image-based interfaces interact via their ndarray views.",3 DM-2417,"Data loader script crashes trying to create chunk table","Vaikunth discovered a bug in data loader when trying to load a data into Object table: {noformat} [CRITICAL] root: Exception occured: Table 'Object_7480' already exists Traceback (most recent call last): File ""/usr/local/home/vaikunth/src/qserv/bin/qserv-data-loader.py"", line 318, in sys.exit(loader.run()) File ""/usr/local/home/vaikunth/src/qserv/bin/qserv-data-loader.py"", line 254, in run self.loader.load(self.args.database, self.args.table, self.args.schema, self.args.data) File ""/usr/local/home/vaikunth/src/qserv/lib/python/lsst/qserv/admin/dataLoader.py"", line 171, in load return self._run(database, table, schema, data) File ""/usr/local/home/vaikunth/src/qserv/lib/python/lsst/qserv/admin/dataLoader.py"", line 209, in _run self._loadData(database, table, files) File ""/usr/local/home/vaikunth/src/qserv/lib/python/lsst/qserv/admin/dataLoader.py"", line 586, in _loadData self._loadChunkedData(database, table) File ""/usr/local/home/vaikunth/src/qserv/lib/python/lsst/qserv/admin/dataLoader.py"", line 653, in _loadChunkedData self._makeChunkAndOverlapTable(conn, database, table, chunkId) File ""/usr/local/home/vaikunth/src/qserv/lib/python/lsst/qserv/admin/dataLoader.py"", line 727, in _makeChunkAndOverlapTable cursor.execute(q) File ""build/bdist.linux-x86_64/egg/MySQLdb/cursors.py"", line 176, in execute if not self._defer_warnings: self._warning_check() File ""build/bdist.linux-x86_64/egg/MySQLdb/cursors.py"", line 92, in _warning_check warn(w[-1], self.Warning, 3) Warning: Table 'Object_7480' already exists {noformat} It looks like I did not do enough testing after my recent improvement in creating chunk tables. It tries to create the chunk table with ""CREATE TABLE IF NOT EXISTS ..."" but that actually generates ""warning exception"" on mysql side when table is already there. Need to catch this exception and ignore it.",1 DM-2419,"Implement authentication mechanism for worker management service","We need some reasonable security for access to new worker management service. It should be lightweight and not depend on complex things that require infrastructure. Something based on a shared secret should be adequate for our immediate needs and likely for the long term.",4 DM-2420,"Document API for worker management service","New worker management service exposes its API as an interface to RESTful web service. Many or all ""methods"" will be wrapped into some sort of Python API, but it would still be useful to document every web service ""methods"" independently. There is basic documentation in the design document (https://dev.lsstcorp.org/trac/wiki/db/Qserv/WMGRDesign), this needs to be extended with detailed description of what those methods do and what kind of data they accept and return. This story involves selecting the right tool.",4 DM-2421,"Improve support for Python modules in Scons","it seems we have two tools to manage python modules: - site_scons/pytarget.py and - site_scons/site_tools/pymod.py (grep for InstallPythonModule) used by admin tools. We could unify this, isn't it?",1 DM-2423,"Weighting in photometric calibration is incorrect","Dominique points out that the zero point calibration uses errors not inverse errors to calculate the zero point. git annotate reveals: bq. 24c9149f python/lsst/meas/photocal/PhotoCal.py (Robert Lupton the Good 2010-12-13 05:03:12 +0000 353) return np.average(dmag, weights=dmagErr), np.std(dmag, ddof=1), len(dmag) Please fix this. At the same time, we should add a config parameter to soften the errors. ",1 DM-2424,"Create transitional duplicate of Span","One challenge in switching from {{PTR(Span)}} to {{Span}} in {{Footprint}} is that Swig won't generate wrappers for {{std::vector}} (or any other container) if {{%shared_ptr(Span)}} is used anywhere else in the codebase. So, to allow both the old {{Footprint}} class and the new {{SpanRegion}} to coexist (temporily), we need to have two {{Span}} classes, one wrapped with {{%shared_ptr}} and one wrapped without it. Since we don't want to disrupt the old {{Footprint}} class yet, we should call the new Span something else, and make it the one that's wrapped without {{%shared_ptr}}. This ticket can be considered complete once we have a unit test demostrating a usable Swig-wrapped {{std::vector}} while all old {{Footprint}} tests continue to pass.",1 DM-2425,"Implement SpanRegion core functionality","Implement the core of the SpanRegion class, as prototyped in RFC-37. This includes the following: - The private implementation object and copy-on-write utilities (see Schema for an example of copy-on-write, but note that SpanRegion's implementation object can be private, while Schema's is not). - All STL container methods and typedefs, and their Pythonic counterparts. - All constructors and assignment operators, except for SpanRegionBuilder. This includes the ability to detect and fix overlapping Spans. - All simple accessors. - {{isContiguous()}} - The shift and clip methods.",6 DM-2426,"Implement SpanRegion+ellipse operations","Implement the following SpanRegion operations: - Construct from an ellipse - note geom::ellipses::PixelRegion; this should do most of the work. - Compute centroid - see old Footprint implementation - Compute shape (quadrupole moments) - see old Footprint implementation One complication here is that this will introduce a circular dependency between afw::geom and afw::geom::ellipses. That's easy to address at the C++ level, but it's tricky in Python (which package imports the other?) I'll be emailing dm-devel shortly to start a discussion on how to address this problem.",2 DM-2427,"Implement SpanRegion applyFunctor methods","Implement methods that apply arbitrary functors to pixels within a SpanRegion, as described on RFC-37. The only tricky part of this implementation will be the ""traits"" classes that allow different target objects to interpreted differently. I'd be happy to consult on this; I have a rough idea in my head, but it needs to be fleshed out.",3 DM-2429,"Add aperture corrections to meas_extensions_photometryKron","When transitioning {{meas_extensions_photometryKron}} to the new measurement framework, aperture correction was omitted pending the completion of DM-85. It needs to be re-enabled when that epic is complete.",1 DM-2430,"Make qserv server-side log messages more standard","Qserv server-side Python logging appears to mostly use a common format: ""{{%(asctime)s %(name)s %(levelname)s: %(message)s}}"". It also mostly uses a common date format: ""{{%m/%d/%Y %I:%M:%S}}"". But I see instances of: * ""{{%(asctime)s %(levelname)s %(message)s}}"" * ""{{%(asctime)s - %(name)s - %(levelname)s - %(message)s}}"" * ""{{%(asctime)s \{%(pathname)s:%(lineno)d\} %(levelname)s %(message)s}}"" * and now, after DM-2176, ""{{%(asctime)s \[PID:%(process)d\] \[%(levelname)s\] (%(funcName)s() at %(filename)s:%(lineno)d) %(name)s: %(message)s}}"" Unless these are used in very different contexts, it will aid automated log processing for them to be more standardized. In addition, the date format is unacceptable as it does not use RFC 3339 (ISO8601) format and does not include a timezone indicator (which means the default {{datefmt}} is insufficient). This must be fixed. See also DM-1203.",1 DM-2431,"Fork GREAT3 sim code and integrate with LSST stack","Get the GREAT3 simulation code running with LSST-provided third-party packages of Python, GalSim, etc, and figure out where we're going to put our modified scripts on GitHub: - Do we just put things in a fork of the great3 repo, or do we have other repos layered on top of a fork of the great3 repo? (I think probably the latter, but we should determine how many repos, and for what purposes.) - Where in GitHub space do we put them (lsst? lsst-dm? user spaces?) This is one of several issues that together will replace DM-1132 (which was just a planning stand-in for these more detailed issues).",4 DM-2432,"Increase postage stamp size in simulation scripts","The GREAT3 simulations have a fixed postage stamp size (though this may differ between branches). A first step at modifying the simulation scripts to meet our needs would be to try to change the postage stamp. This is one of several issues that together will replace DM-1132 (which was just a planning stand-in for these more detailed issues).",2 DM-2433,"Create simulation script with different constant PSF per galaxy.","Modify the GREAT3 simulation scripts to create a branch in which each galaxy gets a different constant PSF, rather than one constant PSF per subfield or a spatially-varying PSF that spans multiple subfields. This could be done by modifying the control/ground/constant branch or the variable-psf/ground/constant branch, or creating an entirely new branch, or anything else (since we don't actually need multiple branches in our simulations). At this point, the source of the PSFs doesn't really matter - as long as we have a class that can provide a different one to every image. This is one of several issues that together will replace DM-1132 (which was just a planning stand-in for these more detailed issues).",6 DM-2434,"Draw simulated PSFs from a library of on-disk files","Modify the simulation code to draw PSFs at random from a library of on-disk files (whose format and on-disk layout should be specified here). The PSFs chosen should be deterministic via a random number generator seed specified via config. This is one of several issues that together will replace DM-1132 (which was just a planning stand-in for these more detailed issues).",4 DM-2435,"Reading an Exposure from disk aborts if the Psf is of an unknown type","Attempting to read an Exposure (in this case via the butler) fails if the PSF class isn't available. An exception would be reasonable, but an assertion failure is not. Running the attached script on tiger-sumire with bq. setup python anaconda; setup -T v10_1_rc2 lsst_apps; setup -j distEst -t HSC; setup -j -r ~/LSST/obs/subaru {code} WARNING: Could not read PSF; setting to null: PersistableFactory with name 'PsfexPsf' not found, and import of module 'lsst.meas.extensions.psfex' failed (possibly because Python calls were not available from C++). {0}; loading object with id=4, name='PsfexPsf' {1}; loading object with id=28, name='CoaddPsf' {2} python: src/table/io/InputArchive.cc:109: boost::shared_ptr lsst::afw::table::io::InputArchive::Impl::get(int, const lsst::afw::table::io::InputArchive&): Assertion `r.first->second' failed. Aborted {code}",1 DM-2436,"Cherry-pick ""fix makeRGB so it can replace saturated pixels and produce an image"" from HSC","HSC-1196 includes fixes and test cases for {{afw}}. After review on HSC, they should be checked/merged to LSST.",1 DM-2437,"Port HSC-side functionality to allow showCamera to display real data via the butler","One of the things that exists on the HSC side of things but not LSST is the ability to use showCamera to create full-focal-plane mosaics. Please convert the code to run with the new cameraGeom Not only is this generically useful, but it's part of the effort required to make the DM-side visualisation work for the Camera group ",4 DM-2439,"iRODS test: Replicate data between servers","A fundamental feature of using iRODS would be to prevent file loss/corruption incidents by replicating data to different physical servers, possibly in geographically disparate locations. We verify that we can replicate data within out test zone/grid.",2 DM-2440,"iRODS test: Virtual collection ","iROD manages data as a 'virtual collection', that is, one can have a single logical/virtual view of a collection of files (the appearance of a single file system/tree) while the data with the collection is stored on separate physical servers. We demonstrate this by creating a collection with data targeted/uploaded to different physical resources.",2 DM-2441,"iRODS test: Register data in place","In our first tests of iRODS, we have used ""iput"" to load data into iRODS cache spaces (the iRODS Vault). For large collections already in a well known location on a server, one may want to leave the data in place but still manage it with iRODS. To do this one can use ""ireg"" to register the data with IRODS without the upload process.",2 DM-2442,"iRODS usage, devel survey","Read up on current IRODS usage and development track. ",3 DM-2444,"Fix and test CheckAggregation","{code:C++} class CheckAggregation { public: CheckAggregation(bool& hasAgg_) : hasAgg(hasAgg_) {} inline void operator()(query::ValueExpr::FactorOp const& fo) { if(!fo.factor.get()); {code} - return is missing here. .get() is not needed, shared_ptr is like regular pointer which is convertible to bool, so whole thing should probably be: if (! fo.factor) return; - We should have a unit test to show us we have problem here ",4 DM-2445,"Fix query ""SELECT * FROM Object o, Source s WHERE o.objectId = s.objectId AND o.objectId = 390034570102582 AND o.latestObsTime = s.taiMidPoint""","Next query fails: {code:bash} mysql --host=127.0.0.1 --port=4040 --user=qsmaster qservTest_case01_qserv -e ""SELECT * FROM Object o, Source s WHERE o.objectId = s.objectId AND o.objectId = 390034570102582 AND o.latestObsTime = s.taiMidPoint"" {code} It seems there's several problems here: * objectId field is duplicated, zookeeper could be used to know all the fields involved by * in a query, but then it has to know each columns. * subChunkId and chunkId are also duplicated, this isn't the case in the plain-mysql query. This duplicated columns prevent the creation of the result table on the czar. ",6 DM-2447,"v10_1_rc2 build test","Test v10_1_rc2 + tickets/DM-2303 on el6, el7, fedora 21, ubuntu 12.04, & ubuntu 14.04. Results to be reported in http://ls.st/faq .",1 DM-2448,"Write additional test for duplicate fields check","Alongside: {code:C++} BOOST_AUTO_TEST_CASE(getDuplicateAndPosition) {code} Add test for: - no duplicate strings - triplicate - more than one string duplicated and alongside: {code:C++} BOOST_AUTO_TEST_CASE(SameNameDifferentTable) {code} test more than one duplicated column, and a column duplicated more than twice.",3 DM-2450,"Fix cmsd-server logger configuration","cmsd-server logger configuration is incorrect: see cmsd.log on Qserv worker: {code:bash} Plugin loaded unreleased QservOssGeneric unknown from osslib libxrdoss.so log4cxx: Could not instantiate class [org.apache.log4j.XrootdAppender]. log4cxx: Class not found: org.apache.log4j.XrootdAppender log4cxx: Could not instantiate appender named ""XrdLog"". log4cxx: No appender could be found for logger (QservOss). log4cxx: Please initialize the log4cxx system properly. QservOss (Qserv Oss for server cmsd) ""worker"" {code}",4 DM-2451,"Fix interface between QservOss and new cmsd version","QservOSS gives an error when attempting to run queries on the worker from the czar. Error log snippet: {code} QservOss (Qserv Oss for server cmsd) ""worker"" 150331 16:06:17 9904 Meter: Unable to calculate file system space; operation not supported 150331 16:06:17 9904 Meter: Write access and staging prohibited. ------ worker@lsst-dbdev3.ncsa.illinois.edu phase 2 server initialization completed. ------ cmsd worker@lsst-dbdev3.ncsa.illinois.edu:36050 initialization completed. {code} ",1 DM-2452,"Replace toString() function","See [~salnikov] comment: Fabrice, anything is possible in C++, if you can define toString() for vectors it should also be possible to define some other construct to format vector into a stream :) My objection to toString() is based on couple of of observations: most of the time in our code converting complex objects to string is done to push them to streams or to logging system (logging is also usually based of streams) methods like toString() are usually implemented using temporary streams. So if you write code like cout << toString(vector) or LOGF_DEBUG(""vector: %1%"" % toString(vector)) it is very inefficient because it creates temporary stream and temporary string(s). To make it more efficient you have to define operator<<() which is implemented without using toString(). Then you could implement toString() based on operator<<() but I'd argue that you should avoid it. In case you really need to convert to string there semi-standard tools which already do the same for types that have operator<< defined (like boost::lexical_cast), but again most of the time you only need operator<< as you don't want to mess with strings. If you want to know how to implement operator<< for vector (or any container) here is the sketch of what I would do (there might be simpler ways): {code:c++} namespace detail { template struct _ContInserterHelper { const Cont& cont; }; template std::ostream& operator<<(std::ostream& out, const _ContInserterHelper& cins) { out << ""[""; const Cont& cont = cins.cont; // this is container itself // print container elements with separators return out << ""]""; } } template detail::_ContInserterHelper ContInserter(const Cont& cont) { return detail::_ContInserterHelper{cont}; } {code} And after that you can do: {code:c++} std::vector v; std::cout << ContInserter(v); {code} And this has no overhead or any temporary objects created. ",3 DM-2454,"investigate github oauth integration for jenkins ","We need a means of authenticating and authorizing users to interact with the CI system. The current seem of using an htpasswd file with buildbot is a hassel both for end user and administratively. Jenkin's has support for ldap and there is a plugin available for github oauth. Administratively, and it terms of reliability, it may make more sense to be coupled with github than a a new DM or the exist LSST LDAP instance.",7 DM-2455,"uncaught exceptions in GaussianFlux","{{SdssShapeAlgorithm::computeFixedMomentsFlux}}, which is used to implement {{GaussianFlux}}, now throws an exception when the moments it is given are singular. That shouldn't have affected the behavior of {{GaussianFlux}}, as it contains an earlier check that should have detected all such bad input shapes. But that doesn't seem to be the case: we now see that exception being thrown and propagating up until it is caught and logged by the measurement framework, resulting in noisy logs. We need to investigate what's going wrong with these objects, and fix them, which may be in {{SdssShape}} or in the {{SafeShapeExtractor}} {{GaussianFlux}} uses to sanitize its inputs.",1 DM-2456,"Participate in April design process","Most work here was with designing firefly tools API related details.",8 DM-2457,"Prepare firefly for GitHub",NULL,8 DM-2458,"Finsh pushing firefly to GitHub",NULL,3 DM-2459,"Begin actual conversion of parts of firefly to pure javascript",NULL,10 DM-2460,"Develop next gen Firefly JavaScript API Tools",NULL,10 DM-2461,"Develop external http api that can control Firefly viewer",NULL,14 DM-2462,"Implement client side of mask layers in FITS image Viewer",NULL,20 DM-2463,"Prepare v10_1 release candidate","Candidate is v10_1_rc2 based on EUPS tag b949",6 DM-2466,"lsstsw ./bin/deploy needs LSSTSW set to install products in the right place","I cloned lsstsw into ~/Desktop/templsstsw and cd'd into it and typed ./bin/deploy and was shocked to find it installed everything into ~/lsstsw, leaving an unsable mess: some files were in templsstsw and some in ~/lsstsw. The short-term workaround is to manually set LSSTSW before running ./bin/deploy, but this should not be necessary; bin/deploy should either set LSSTSW or not rely on it. I don't recall this problem with earlier versions of lsstsw; I think this is a regression. For now I updated the instructions at https://confluence.lsstcorp.org/display/LDMDG/The+LSST+Software+Build+Tool but I look forward to being able to revert that change.",1 DM-2467,"Implement stitching multiple patches across tract boundaries in a coadd v2","* Find region that returns multiple tractPatchLists for testing. * Request region via central point (RA, Dec) with width and height definable in arcseconds and pixels. * May be extend web interface to other data sets, and/or good seeing SkyMaps. ",8 DM-2468,"Qsev Documentation",NULL,10 DM-2469,"Turn on C++ 11 flag for Qserv",NULL,4 DM-2470,"Handle all exceptions coming from worker",NULL,7 DM-2475,"Build 2015_04 Qserv release","See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe.",1 DM-2477,"Design API and RFC design","Use the HSC implementation of the base class as a point of reference for designing an integrated Approximate and Interpolate class. The design take into account Chebyshev, spline, and Gaussian process mechanisms. Want to take into consideration client code. I.e. it shouldn't make current consumers more complicated (background and aperture correction to name two). RFC the designed API.",8 DM-2478,"Edit background class","Make fixes to background class to use new approximate/interpolate class.",10 DM-2479,"Fix-up any code that uses approximate/interpolate","The background matcher is one area where the approximate/interpolate class will be used. This story will find all places (including examples and unit tests) where the old approximate/interpolate mechanisms are used and update them to use the new interface.",4 DM-2480,"Delete old approximate/interpolate classes","Once all updates are done to code and unit tests pass with the new approximate/interpolate interface, the old ones should be completely removed.",2 DM-2484,"Justify level of staff at La Serena, to the level of justifying for office space","help with the specifications for the buildings in La Serena. The request si enough prove to justify office space for the DM administrators. and for other staffing needed for DM. This work will span two weeks and is due this Thursday April 9. This story is for the orienting work - -kickoff phone call.",1 DM-2485,"recieve and begin to process document from SET about scalability of CEPH","IN the context of ISL investigations into stogie systems the SET group has produced a document that goes into the scaling of the meta data services. The concern is that there is a central meta data service ins CEPH. Began to process this analysis and to think about feasibility of testing program.",1 DM-2486,"management for week March 30.","Investigated invoicing fro storage condo -- appears to be annual fee, OK by Jeff. Investigated attaching effort breakdown to invoke -- this seems hard as U of I invoicing occurs at quite a distance (procedural) distance from the NCSA business office. Decided to look at improvements in recording effort in Jira so as to be able to generate report. -- Capture all actuals. Business office transition support is transitioning from Matt S. to new person. Review AMCL sides, kept tradition generating exponentially more comments, but reduced the exponent. Process to bill out effort applied to project, but not in standing assignments in the staffing plan. Internal strategy meeting about agenda items w.r.t VAO given Rap Plante is leaving NCSA. Prep for DM leadership meeting -- synergies at NCSA. ",4 DM-2487,"security weekly meeting ","met with the ISO, looking for ways to more actively engage. Idea was to focus on the SCADA enclave, and the need was to engage with German Etc",1 DM-2489,"Draft SCADA security plan",NULL,10 DM-2491,"Initial survey of Datacat for LSST ","Jacek, Brian Van Klaveren have sent along some initial overview/description of their work on Datacat; https://confluence.slac.stanford.edu/display/~bvan/LSST+Datacat+Overview We start examining this in the context of our studies of managing data collections at NCSA.",1 DM-2492,"shapelet unit tests attempts to access display on failure","When tests/profiles.py tests fail, they attempt to create live plots without checking for any variables that indicate that the display should be used. These plots should be disabled, as they obscure the real error when the display is not available.",1 DM-2493,"prepare v10_1_rc3 release candidate","Need for rc3 identified ",10 DM-2497,"Fix g++ 4.9 return value implicit conversion incompato","g++ 4.9 enforces the ""explicit"" keyword on type conversion operators in return value context. This mean bool checkers along the lines of bool isValidFoo() { return _smartPtrFoo; } require an explicit cast to compile under g++ 4.9 with -std=c++0x. There were a handful of these in our code; found and fixed.",1 DM-2498,"run jenkins builds on multiple platforms","Demonstrate a Jenkins build matrix running lsstswBuild.sh on a number of platforms including; el6, el6, f21, u12.04, & u14.04.",28 DM-2500,"Mountain - Base fiber path design and installation method","Design path and installation method for Mountain - Base fiber cable. Path will run from Cerro Pachon to Cerro Tololo to AURA gate. Installation method will define where the fiber cable will be on poles or underground.",40 DM-2502,"Improve db.createTable","DM-2417 revealed that the current implementation of createTable in db module behaves differently that mysql: mysql will issue a warning if table exists, and db module will fail with an error. We should make the db behave similarly to how mysql behaves. ",2 DM-2503,"Doxygenize db","The db module needs to be doxygenized.",1 DM-2504,"Optimize support for many identical database schemas - design","It is likely we will have 100s or 1000s of identical databases (identical in terms of schema). It'd be good to not repeat the schema information in metaserv. This ticket include coming up with a plan how to implement it.",1 DM-2505,"Optimize support for many identical database schemas - impl","It is likely we will have 100s or 1000s of identical databases (identical in terms of schema). It'd be good to not repeat the schema information in metaserv. This ticket include implementing a clean solution, proposed through DM-2504",2 DM-2506,"Document structure of our custom ddl ascii schema","Need to better document what is supported / accepted by schemaToMeta.py. We are currently relying on cat/sql/baselineSchema.sql as the guide.",2 DM-2507,"Information exchange between processes - research","We need to identify a reliable and fast way to exchange information between processes (for example, cmsd and xrootd). This story involves understanding key requirements (structures, scale), and researching what mechanism would be best). Deliverable: short narrative describing key requirements, and proposed mechanism, including a sketch of the design.",4 DM-2508,"Information exchange between processes - implementation","Implement system for information exchange between cmsd and xrootd, per instructions in DM-2507",8 DM-2510,"Research feasibility of using SQLite as backend to the db module","This story involves plugging in SQLite and dealing with issues that arise as a result of using SQLite in places that depend on the db module.",10 DM-2511,"The distance field of match lists should be set","The meas_astrom AstrometryTask returns a match list that has distance = 0 for all elements. Neither the matcher nor the WCS fitter are setting this field, and both ought to.",2 DM-2512,"FY17 Integrate Web Services with NCSA Authentication System","We need to integrate Data Access Web Services with Authentication mechanisms used by NCSA.",40 DM-2513,"W16 Improvements to db","Improvements to Db wrapper.",31 DM-2514,"Migrate to new WBS for 02C.06","Migrate to the new WBS structure for 02C.06. Work include: * revisiting wbs assignment for all epics * updating [S15 planning 4 DB team|https://confluence.lsstcorp.org/display/DM/S15+planning+4+DB+team] * updating ldm-240 spreadsheet * updating associated budget accounts * tweaking [build-ldm240.py|https://github.com/jbecla/experimental/blob/master/build-LDM-240.py]",1 DM-2515,"Catch ""address in use""","I noticed when running integration tests, it failed with the error pasted below. It'd be good to catch it and print something useful. I am not entire sure what port number is in use, and what to kill... {code} File ""/usr/local/home/becla/stack_201502/repo/qserv/bin/qservWmgr.py"", line 89, in sys.exit(main()) File ""/usr/local/home/becla/stack_201502/repo/qserv/bin/qservWmgr.py"", line 85, in main app.run(host) File ""/usr/local/home/becla/stack_201502/Linux64/anaconda/2.1.0/lib/python2.7/site-packages/flask/app.py"", line 772, in run run_simple(host, port, self, **options) File ""/usr/local/home/becla/stack_201502/Linux64/anaconda/2.1.0/lib/python2.7/site-packages/werkzeug/serving.py"", line 710, in ru n_simple inner() File ""/usr/local/home/becla/stack_201502/Linux64/anaconda/2.1.0/lib/python2.7/site-packages/werkzeug/serving.py"", line 692, in in ner passthrough_errors, ssl_context).serve_forever() File ""/usr/local/home/becla/stack_201502/Linux64/anaconda/2.1.0/lib/python2.7/site-packages/werkzeug/serving.py"", line 486, in ma ke_server passthrough_errors, ssl_context) File ""/usr/local/home/becla/stack_201502/Linux64/anaconda/2.1.0/lib/python2.7/site-packages/werkzeug/serving.py"", line 410, in __ init__ HTTPServer.__init__(self, (host, int(port)), handler) File ""/usr/local/home/becla/stack_201502/Linux64/anaconda/2.1.0/lib/python2.7/SocketServer.py"", line 419, in __init__ self.server_bind() File ""/usr/local/home/becla/stack_201502/Linux64/anaconda/2.1.0/lib/python2.7/BaseHTTPServer.py"", line 108, in server_bind SocketServer.TCPServer.server_bind(self) File ""/usr/local/home/becla/stack_201502/Linux64/anaconda/2.1.0/lib/python2.7/SocketServer.py"", line 430, in server_bind self.socket.bind(self.server_address) File ""/usr/local/home/becla/stack_201502/Linux64/anaconda/2.1.0/lib/python2.7/socket.py"", line 224, in meth return getattr(self._sock,name)(*args) socket.error: [Errno 98] Address already in use {code}",1 DM-2518,"Add a CFHT-based post-build integration test to the sandbox build","From [~boutigny] I have installed some simple stack validation tools working on CFHT data in {{/lsst8/boutigny/valid_cfht}} Here is the content of the README file : ------------------------------------------------------------------------------------------------------------------------ This directory contains a set of utilities to validate a stack release with CFHT data At the moment, only validation plots for the astrometry are produced Directories : ------------- rawDownload : contain raw CFHT images (flat, dark, bias, fringe,... corrected) reference_plots : contain reference plots corresponding to the best results obtain so far. Files : ------- setup.cfht : stack environment setup valid_cfht.sh : run processCcd taks on the cfht images valid_cfht.sh init : create the input/ouput directories, ingest raw images and run processCcd valid_cfht.sh : without the ""init"" argument, runs processCcd assuming that the directory structure exists and that the raw images have been ingested. valid_cfht.py : run some analysis on the output data produced by valid_cfht.sh processConfig.py : configuration parameters for processCcd run.list : list of vistits / ccd to be processed by processCcd Requirements : -------------- obs_cfht : tickets/DM-1593 astrometry_net_data : SDSS_DR9 reference catalog corresponding for CFHT Deep Field #3 ------------------------------------------------------------------------------------------------------------------------ Basically it produces a set of plots stored in a png image that can be compared to a reference plot corresponding to the best results obtained so far with stack_v10_0 I hope that this is useful. Just be careful that I wrote these scripts with my own ""fat hand full of fingers"" and that it is just basic code from a non expert. If it is useful, I can certainly add more plots to validate the psf determination, photometry, etc. Comments, suggestions and criticisms are very welcome.",1 DM-2519,"Check for Qserv processes at configuration tool startup","Configuration tool has to check for Qserv processes before removing configuration directory (which may contains init.d scripts for these running processes)",4 DM-2520,"Proof of concept Python APIs to access Firefly components","The pipeline needs to visualize the images using Firefly. We want to provide a few Python APIs for proof of concept that we could do this in Python and IPython notebook. ",6 DM-2521,"Update repo.yaml for first set of Sims Stash repo moves","The repos.yaml file needs to be updated with correct repository locations once SIM-1074 is completed.",1 DM-2522,"Implement distributed database deletion","Implement database deletion based on the process defined in DM-1396. Need to deal with situations like worker is offline - might need some infrastructure e.g., running something in background to act when affected workers come back online. Deliverable: a demonstration of system that deletes a distributed database: user issues ""drop database x"" and all copies of that database on all workers, all replicas of all chunks are deleted. It should be possible to ""create database x"" at any time later.",6 DM-2523,"Ensure we can delete/create table with the same name","Test / ensure that we can create a table with the same name as the table we just deleted.",4 DM-2524,"Margaret's mgmt. activities in March","Catch-all story for LOE activities in March 2015. 20 working days - 7 days vacation = 13 actual days 13 days * 2 SP/day * .5 FTE = 13 SP.",13 DM-2526,"Tom Durbin on board as a consultant for the Base site data center.","Prep/attend/follow through for meeting with Tom Durbin, the facility manager of the National Petascale Computing Facility, to discuss participating as consultant to the project as it finds a design contractor, and as the design matures. ",1 DM-2528,"Mgt activity summary for week of April 6","- Made inquires about the status of materials contracting - ball in AURA's court. - Prepare for visit to Lyon. Consult with atoll, researched collaborative structures, articulated and vetted hardware process, made some slides for the visit. - Spent time thinking about VO protocols and such in prep for the DM F2F discussion. - Edited Job descriptions for the ADS department ,who will recruit for our systems engineers to include LSST, and LSST concerns. - Management / leading by walking around N.B. Margaret in CAM training (or associated travel) Tu-F. N.B Don was off 1 1/2 days ",3 DM-2530,"Resolve outgoing port issues on Blue Waters/Cray systems ","pro data system scaling tests on cray system were limited by the number of outgoing ports on a cray node. The limitation had been ~20 ports, participated in Tests of new system software,limit relaxed to at least ~2000 in tests. Likely greater.",3 DM-2532,"discussed the request from the SUI group for authentication guidance ","responded to ticket from Jacek, on behalf, I think of the SUI group asking for guidance on authentication at NCSA. So far, consulted with Alex Withers, contemplating the extent of policies so far (not much) an authentication mechanism worth investigtaing and likely policies. Drew figure for discussion, wrote up in hip chat.",1 DM-2533,"Remove version attribute from Schema","Remove the Schema attribute and its getters and setters. This change won't be something we can merge to master on its own, as it doesn't provide backwards-compatible FITS reading that will added in future tasks.",1 DM-2534,"Rewrite afw::table FITS reading to be more flexible","In order to support backwards-compatible FITS table reading, we need to break the current assumption that everything we need to know about how to read a Record from a FITS file is contained in the Record's Schema. This issue involves that refactoring, without actually adding the backwards compatibility support.",4 DM-2535,"Backwards compatibility for reading compound fields from FITS","Read old-style afw::table compound fields in as scalar fields, using the new FunctorKey conventions.",2 DM-2536,"Backwards compatibility for reading slots and measurements from FITS","Rename fields to match the new slot and measurement naming conventions.",2 DM-2537,"Contextual error handling","There are cases when an empty result might have different errors than the top error, and it would be good to unwrap the context in which the error occured. Example: GET /meta/v0/db/L3/joe_myDb/tables/Object, the result might be empty because the database does not exist, or the v0 is not a supported version, etc.",4 DM-2538,"RESTful python client","Develop basic abstractions for restful apis in a python client",3 DM-2541,"Research Ceph file system","Research Ceph as possible networked filesystem for LSST usage to replace NFS. Estimate spending 10-20 hours of work with result being a wiki page of suggestions, limitations, etc. (Implementation will be a different task, presuming we want to implement.)",6 DM-2543,"Python APIs for Firefly ","We need Python APIs to interface with Firefly visualization components. This is the first set of many functions. ",8 DM-2544,"ctrl_events build issue","Had a problem where ctrl_events was having build issues.",1 DM-2545,"LaTeX support in Doxygen broken","LaTeX markup in Doxygen documentation ought to be rendered properly for display in HTML. It isn't: it's just dumped to the page as raw text. See, for example, [the documentation for {{AffineTransform}}|https://lsst-web.ncsa.illinois.edu/doxygen/xlink_master_2015_04_15_07.01.28/classlsst_1_1afw_1_1geom_1_1_affine_transform.html#details].",1 DM-2546,"Host.cc doesn't find gethostname and HOST_NAME_MAX under el7","el7 gives an error that it can't find HOST_NAME_MAX.",1 DM-2547,"Fix again interface between QservOss and new cmsd version","QservOSS gives an error when attempting to run queries on the worker from the czar. Error log snippet: {code} QservOss (Qserv Oss for server cmsd) ""worker"" 150331 16:06:17 9904 Meter: Unable to calculate file system space; operation not supported 150331 16:06:17 9904 Meter: Write access and staging prohibited. ------ worker@lsst-dbdev3.ncsa.illinois.edu phase 2 server initialization completed. ------ cmsd worker@lsst-dbdev3.ncsa.illinois.edu:36050 initialization completed. {code} ",8 DM-2549,"The string repr of Coord should show the coordsys and angles in degrees","The default string representation of Coord (e.g. std::cout << coord in C++ and str(coord) in Python) is to show class name and a pair of angles in radians. It would be much more useful if the default display showed the angles in degrees, as that is what people are used to. Also, it would be very helpful if the display included the name of the coordinate system. This is especially needed for the base class, as it is quite common to get shared_ptr to Coord and have no idea what coordinate system it is. At present there is a lot of code that unpacks the angles and explicitly displays them as degrees to get around this problem. But it seems silly to have to do that.",2 DM-2551,"ANetAstrometryTask's debug doesn't fully work","{{ANetAstrometryTask}}'s debug code calls (deprecated) method {{Task.display}}, which raises an AttributeError on this coce: {code} try: sources[0][0] except IndexError: # empty list pass except (TypeError, NotImplementedError): # not a list of sets of sources {code} ",1 DM-2552,"xrootd can't be started via ssh","{code:bash} qserv@clrinfopc04:~/src/qserv$ ssh localhost -vvv ""~qserv/qserv-run/2015_02/etc/init.d/xrootd start"" ... debug3: Ignored env _ debug1: Sending command: ~qserv/qserv-run/2015_02/etc/init.d/xrootd start debug2: channel 0: request exec confirm 1 debug2: callback done debug2: channel 0: open confirm rwindow 0 rmax 32768 debug2: channel 0: rcvd adjust 2097152 debug2: channel_input_status_confirm: type 99 id 0 debug2: exec request accepted on channel 0 Starting xrootd.. debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0 debug2: channel 0: rcvd eow debug2: channel 0: close_read debug2: channel 0: input open -> closed {code} Here ssh command freeze, it is possible to lauch xrootd with this (example) script: {code:bash} set -e set -x . /qserv/run/etc/sysconfig/qserv export QSW_XRDQUERYPATH=""/q"" export QSW_DBSOCK=""${MYSQLD_SOCK}"" export QSW_MYSQLDUMP=`which mysqldump` QSW_SCRATCHPATH=""${QSERV_RUN_DIR}/tmp"" QSW_SCRATCHDB=""qservScratch"" export QSW_RESULTPATH=""${XROOTD_RUN_DIR}/result"" export LSST_LOG_CONFIG=""${QSERV_RUN_DIR}/etc/log4xrootd.properties"" eval '/qserv/stack/Linux64/xrootd/xssi-1.0.0/bin/xrootd -c /qserv/run/etc/lsp.cf -l /qserv/run/var/log/xrootd.log -n worker -I v4 &' echo ""SCRIPT STARTED"" {code} and the same problem occurs. So the problem seems to be with xrootd, and not the startup scripts. ",5 DM-2554,"Remove most compound fields from afw::table","Remove all Point, Moment, Coord, and Covariance compound fields. Array fields should be retained for now; it's not clear if we want to remove it or not, or how to handle variable-length arrays if we do.",2 DM-2555,"Create and advertise Firefly mailing list","Create an IPAC mailing list for all users of Firefly. Advertise it to the interested communities (including the LSST Camera group) and through the Github site. The mailing list firefly@ipac.caltech.edu has been created and all the interested partied have been subscribed to the list.",1 DM-2556,"Make dbserv async",NULL,5 DM-2557,"Vectorize methods for locating objects on detectors","vectorize _transformSingleSys and _findDetectors in afw.cameraGeom so that the sims_coordUtils method findChipName (which finds the chips that an object lands on) runs faster.",2 DM-2558,"Migrate qserv code to reworked db/dbPool","Migrate code to the new implementation of SQLAlchemy-based Db module, including removal of DbPool.",11 DM-2563,"Chilean Network LOE ",NULL,40 DM-2572,"Addressing File corruption in iRODS 3.3.1","In this issue we examine how file corruption would be detected and repaired with iRODS tools and rules/microservices.",16 DM-2573,"read and understood proposal to consider CAS/crowd system ","the FERMI telescope has an authentication system based on CAS/Crowd. The benefit of the system is that it can be use as an authentication system for both web and command line. Download materials, and acquire the understanding from a review of documentation . Discuss with the ISO, propose discussion for vTony's visit to NCSA (may 21).",1 DM-2574,"management activities for week of April 13","Read proposed ""Hardware"" contract amendment, sent marked up comments to Julie Robinson, U of I contract negotiator. Major points are that Hardware is not descriptive of all purchases needed to fulfill SOW. The procurement approval process needs spelling out. Detailed guidance in comments inserted into contract. Along with M. Gelman met with the NCSA business people to fully understand the U of I invoicing process, and the information in the existing business processes. prior to inventing processes for the supplementing the U of I invoice with the more detailed annotations (hours by WBS) agreed to in the LSST contract. Obtained help from the NSCS IT group. Documented in tow page note. Met concerning seemingly large amount of effort to respond to hip chat take about slowness in the NCSA development system. Miscellaneous and meetings. ",5 DM-2578,"Research BeeGFS file system","Research BeeGFS as possible networked filesystem for LSST usage to replace parts of NFS. Estimate spending 10 hours of work with result being a wiki page of suggestions, limitations, etc. (Any implementation will be a different task, presuming we want to implement.) http://www.beegfs.com/content/ BeeGFS (formerly FhGFS) is a parallel cluster file system, developed with a strong focus on performance and designed for very easy installation and management. If I/O intensive workloads are your problem, BeeGFS is the solution. Likely not good replacement for formal/managed data, but perhaps great option for shared scratch file systems.",5 DM-2579,"Calling AliasMap::get("""") can return incorrect results","It looks like empty string arguments can cause AliasMap to produce some incorrect results, probably due to the partial-match logic being overzealous.",1 DM-2580,"Implement user-friendly template customization","Qserv configuration tool has to be improved to allow developers/sysadmin to easily use their custom configuration files (with custom log level, ...) for each Qserv services. An optional custom/ config file directory will be added, and configuration files templates which will be here will override the ones in the install directory. This should be thinked alongside configuration management inside Docker container.",5 DM-2581,"log4cxx build failure on OS X","[~frossie] writes: {quote} I have a log4cxx failure on a Macp while building lsst_distrib. Attaching file in case someone has any bright ideas for me in the morning {quote}",1 DM-2582,"Research MaxScale as a mysql-proxy replacement","We have been told by Monty that MaxScale is the replacement of the mysql-proxy. Based on DM-2057 the sentiment is that it won't work for our needs. We should very briefly document what our needs are, how we use the proxy now, and if we think MaxScale is not good-enough, say it why, and discuss with Monty and his team.",5 DM-2585,"Purchase of network equipment for use in Chile","jkantor, rlambert",40 DM-2586,"Base LAN Network Design ","Design of the network at the Base to to provide services for the ""tenants"" Telescope, Camera and DM",20 DM-2587,"Base Network LOE",NULL,40 DM-2589,"Design the Network from NCSA to Ampath in Florida",NULL,20 DM-2590,"Comparison of ALMA and Reuna/AURA costs on National link",NULL,4 DM-2591,"Comparison of ALMA summit to base link with LSST",NULL,2 DM-2592,"Remove obsolete hinting code in proxy","Remove now dead code related to sending hints from proxy to czar",1 DM-2593,"Client API for new worker management service","We have new worker management service which has HTTP interface, now we need to provide simple way to access it from Python basically wrapping all HTTP details into simple Python API. ",8 DM-2594,"Change repos.yaml for next set of Simulations Stash repos","The next set of Simulations Stash repository migrations is laid out in SIM-1121.",1 DM-2595,"Symlink data directory at configuration","We decided to introduce symlinks in order to protect data. This is in particular useful when we need to reinstall qserv, but we have valuable, large data set that we want to preserve. This story introduces symlinks to data: when Qserv is reinstalled, only the symlink is destroyed, and the data stay untouched.",5 DM-2597,"Fiber installation on AURA property from Gate to Pachon","AURA and Reuna oversee Telefonica in installing the fiber from the AURA Gate to Cerro Tololo to Cerro Pachon.",20 DM-2599,"afw.Image.ExposureF('file.fits.fz[i]') returns the image in 'file.fits.fz[1]' ","It seems that afwImage.ExposureF ignores the extension number when this is passed on as part of the filename and uses the image in extension number 1. This is not the case with afwImage.MaskedImageF which correctly uses the input extension number passed in the same way. The problem has been checked on OSX Yosemite 10.10.3 with the is illustrated in the following code https://gist.github.com/anonymous/d10c4a79d94c1393a493 which also requires the following image in the working directory: http://www.astro.washington.edu/users/krughoff/data/c4d_130830_040651_ooi_g_d1.fits.fz ",3 DM-2600,"FY18 Integrate DRP with Data Provenance","Integrate the Data Provenance system with the DRP. This includes capturing hardware and software configuration, as well as dependencies between data sets. Deliverable: System capable of capturing provenance for DRP.",54 DM-2601,"'eups distrib install flask -t qserv' fails on Ubuntu 14.04","Qserv now depends on Flask, so this blocks all Qserv install which rely on eups. Comman below works with system-python but not with anaconda: {code} qserv@clrinfoport09:~/stack/EupsBuildDir/Linux64/flask-0.10.1/flask-0.10.1⟫ python setup.py install --home /home/qserv/stack/Linux64/flask/0.10.1 running install Traceback (most recent call last): File ""setup.py"", line 110, in test_suite='flask.testsuite.suite' File ""/home/qserv/stack/Linux64/anaconda/master-g68783b1848/lib/python2.7/distutils/core.py"", line 151, in setup dist.run_commands() File ""/home/qserv/stack/Linux64/anaconda/master-g68783b1848/lib/python2.7/distutils/dist.py"", line 953, in run_commands self.run_command(cmd) File ""/home/qserv/stack/Linux64/anaconda/master-g68783b1848/lib/python2.7/distutils/dist.py"", line 972, in run_command cmd_obj.run() File ""/usr/lib/python2.7/dist-packages/setuptools/command/install.py"", line 73, in run self.do_egg_install() File ""/usr/lib/python2.7/dist-packages/setuptools/command/install.py"", line 82, in do_egg_install cmd.ensure_finalized() # finalize before bdist_egg munges install cmd File ""/home/qserv/stack/Linux64/anaconda/master-g68783b1848/lib/python2.7/distutils/cmd.py"", line 109, in ensure_finalized self.finalize_options() File ""/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py"", line 274, in finalize_options ('install_dir','install_dir') File ""/home/qserv/stack/Linux64/anaconda/master-g68783b1848/lib/python2.7/distutils/cmd.py"", line 298, in set_undefined_options src_cmd_obj.ensure_finalized() File ""/home/qserv/stack/Linux64/anaconda/master-g68783b1848/lib/python2.7/distutils/cmd.py"", line 109, in ensure_finalized self.finalize_options() File ""/usr/lib/python2.7/dist-packages/setuptools/command/install_lib.py"", line 13, in finalize_options self.set_undefined_options('install',('install_layout','install_layout')) File ""/home/qserv/stack/Linux64/anaconda/master-g68783b1848/lib/python2.7/distutils/cmd.py"", line 302, in set_undefined_options getattr(src_cmd_obj, src_option)) File ""/home/qserv/stack/Linux64/anaconda/master-g68783b1848/lib/python2.7/distutils/cmd.py"", line 105, in __getattr__ raise AttributeError, attr AttributeError: install_layout {code}",5 DM-2602,"FY18 Integrate Calibration Pipe with Data Provenance",NULL,26 DM-2603,"FY19 Integrate L3 with Data Provenance","Integrate L3 (images and databases) with Data Provenance.",80 DM-2605,"Package flask dependencies","We packaged flask (see dm-1797) and we are using it via eups, but we have not packaged flask dependencies, and we are still relying on anaconda to get them. This story involve packaging the dependencies.",3 DM-2606,"HSC backport: recent Footprint fixes","This is a backport issue to capture subsequent HSC-side work on features already backported to afw. It includes (so far) the following HSC issues: - [HSC-1135|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1135] - [HSC-1129|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1129] - [HSC-1215|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1215]",2 DM-2607,"W16 Qserv Refactoring","Refactoring of Qserv as found necessary in W16.",60 DM-2608,"FY17 Qserv Refactoring",NULL,100 DM-2609,"FY18 Qserv Refactoring",NULL,100 DM-2611,"API key case study",NULL,1 DM-2612,"Margaret's mgmt. activities in April","- Weekly DMLT phone meetings - Weekly security meetings - Weekly local group meetings - T/CAM training meeting - 1 week w/ travel - Attended leadership meetings about NCSA/UIUC receiving and inventory policies and procedures and grant management for project managers - Prepared slides for AMCL - Updated LDM-240 milestones for FY16+ - Created invoice breakdown (incorrectly!) - Prepared March Technical Progress Report - Prepared Travel Expense Report for T/CAM meeting - Attempted to update S15 Staff Plan in PMCS - Reviewed risk registry with Don - Met with Julie Robinson & AURA to work on/discuss procurement contract amendment - Met with Kaylyn and Alan to discuss invoicing, billing process and timeline, WBS activity code breakdown in MIS - Met with Jay and Kaylyn about staff planning and budget - Met with Nathan to figure out how to track a lot of this EV stuff by downloading to a local database and integrating MIS information - Developed swimlane diagram to understand roles and responsibilities of reporting - Started working on Gantt chart to wrap my head around tracking resource loading and activity progress",31 DM-2613,"weekly liaison with ISO","Discussed ""should we piggyback signing of the LSST AUP with a capability offered by OSG""? with Alex. Additional understanding of Authorization/Authentication.",1 DM-2614,"Management activities for week of april 21","Met with Julie Robinson, the Illinois contract negotiator, w.r.t. aura ""hardware"" contract amendment. re-drafted long paragraph i AURA section, breaking it down into separate items for each party, and addressed what I see a grave flaws so that a discussion could be held. Did work w Margaret do arrange for business proscesst discussion relating to monthly reporting to LSST appended to invoices, basic Interviewed Martin Paegert (one day visit). Further work on other other matters relating to open requisitions of people. Other work on personal matters Responded to comments about the NCSA WBS and overall project WBS not being aligned. on LLDM-240 -- provided example of working one case -- scheduled for next wee'k LT. Misc.",5 DM-2615,"LOE - Week ending 5/1/15","- patch maintenance (kernel, zfs, Intel NIC, esxi NICs) on Thursday",12 DM-2618,"Drop PK on overlap tables in data loader",NULL,2 DM-2619,"Reimplement Data Loader Using Worker Mgmt Service","Current loader depends on ssh, need to switch to the new service, http based.",8 DM-2621,"Add version stamping in czar and ssi service","DM-2547 will introduce compile-time version generation of a header file that has macros defining version strings. Ideally, each running process using qserv code (e.g., czar, cmsd, xrootd, and mysql-proxy), and perhaps one-shot binaries (loader?) would print version information when logging to improve debuggability. DM-2547 focused on the osslib plugin (libxrdoss) for the cmsd. The next important processes are the czar and xrootd (libxrdsvc). This story covers inclusion of version identifiers (w/ commit hash) in the czar and xrootd logs. Hopefully this will end any confusion about versions when reading log files sent from colleagues.",3 DM-2622,"Modify czar to support table deletion","Czar needs to handle table deletion. In practice that means mysql proxy should let DROP TABLE queries through, and czar should modify appropriate table-related metadata structures in CSS. This is part of work proposed in DM-1896. ",6 DM-2623,"Design Basic Watcher","Design watcher, including its interactions with other components (mysql, css, etc). In the near term, the watcher will handle deleting tables and databases.",2 DM-2624,"Implement DROP table in watcher","Implement DROP table using the watcher designed in DM-2623.",1 DM-2625,"Create service for managing watcher","We need to be able to start/stop the watcher implemented through DM-2624. This story involves extending our scripts for starting various qserv services to manage watcher.",1 DM-2627,"Add support for configuring multi-node integration tests","The multi-node integration test software produced through DM-2175 has hardcoded node names. This story will allow user to configure it. Current plan is to pre-set integration test for several different configurations, e.g., single-node, 2-node, 8-node (and maybe eg 24-node), and user would supply node names through a configuration file.",5 DM-2628,"Integration test succeeds when individual tests fail","Integration test behaves strangely, it always succeeds even though there may be tests that fail. Here is what I ge when I run individual case: {noformat} [salnikov@lsst-dbdev4 dm-2617]$ qserv-check-integration.py -i 01 -l ............... 2015-04-28 11:26:12,137 - lsst.qserv.tests.benchmark - ERROR - MySQL/Qserv differs for 4 queries: 2015-04-28 11:26:12,138 - lsst.qserv.tests.benchmark - ERROR - Broken queries list in /usr/local/home/salnikov/qserv-run/2015_04/tmp/qservTest_case01/outputs/qserv: ['0001_fetchObjectById.txt', '0003_selectMetadataForOneGalaxy_withUSING.txt', '0003_selectMetadataForOneGalaxy_classicJOIN.txt', '0003_selectMetadataForOneGalaxy.txt'] 2015-04-28 11:26:12,138 - root - CRITICAL - Test case #01 failed {noformat} But if I run integration test it says everything is OK: {noformat} [salnikov@lsst-dbdev4 dm-2617]$ qserv-test-integration.py ................... ok ---------------------------------------------------------------------- Ran 5 tests in 160.058s OK {noformat} There are actually messages about failed test in the output but you have to look very closely not to miss them. ",1 DM-2629,"Fix build for gcc 4.7.2 and gcc 4.8.2","#include is missing in threadSafe.h",1 DM-2630,"Document configuration tool main use cases","- Document main use case for qserv-configure.py: install Qserv master/worker node with externalized data directory - Hide complex configuration options? {code} Configuration steps: General configuration steps -d, --directory-tree Create directory tree in QSERV_RUN_DIR, eventually create symbolic link from QSERV_RUN_DIR/var/lib to QSERV_DATA_DIR. -e, --etc Create Qserv configuration files in QSERV_RUN_DIR using values issued from meta-config file QSERV_RUN_DIR/qserv-meta.conf -c, --client Create client configuration file (used by integration tests for example) Components configuration: Configuration of external components -X, --xrootd Create xrootd query and result directories -C, --css-watcher Configure CSS-watcher (i.e. MySQL credentials) Database components configuration: Configuration of external components impacting data, launched if and only if QSERV_DATA_DIR is empty -M, --mysql Remove MySQL previous data, install db and set password -Q, --qserv-czar Initialize Qserv master database -W, --qserv-worker Initialize Qserv worker database -S, --scisql Install and configure SciSQL {code} ",3 DM-2631,"Use WebSocket for communication between client and web server, proof of concept","Research and proof of concept code to use Web Socket for two-way communication between client and web server. ",10 DM-2632,"implementation of Web Socket for two-way communication between client and Web server ","Implementation of web socket to be used as the two-way communication method between client and web server. ",10 DM-2633,"refactor the image stretch code for better, simplified organization ",NULL,8 DM-2634,"add new image stretch algorithm to Firefly visualization ","There is a need to include two new stretch algorithms, which are asinh and power law gamma. The algorithm is as follow: * asinh ## input zp: zero point of data mp: maximum point of data dr: dynamic range scaling factor of data. It ranges from 1-100,000 bp: black point for image display wp: white point for image display ## calculate rescaled data value rd = dr *(xPix - zp)/mp ## calculate normalized stretch data value nsd = asinh(rd)/asinh(mp-zp) ## calculate display pixel value dPix = 255 * (nsd-bp)/wp Note: The bp, wp values specify how far outside of the scale data one wants the image to display. By default, setting bp=0 and wp=dr. * power law gamma ## input \br zp: zero point of data mp: maximum point of data gamma: gamma value for exponent ## calculate rescaled data value rd = xPix - zp ## calculate normalized stretch data value nsd = rd^(1/gamma) / (mp0zp)^(1/gamma) ## calculate display pixel data value dPix = 255 * nsd ",8 DM-2635,"Provide a function to return the path to a package, given its name","As per RFC-44 we want a simple function in utils that returns the path to a package given a package name. This has the same API as eups.getProductDir, but hides our dependence on eups, as per the RFC.",2 DM-2636,"Update code to use the function provided in DM-2635","As per RFC-44: update existing code that finds packages using eups.getProductDir or by using environment variables to use the function added in DM-2635",3 DM-2638,"Run large scale tests","Coordinate running large scale tests.",6 DM-2642,"missing dependencies in scons builds","ndarray and afw have some headers generated via m4, and while those are built when the package is installed, if someone just tries to build other targets, they aren't - leading to build failures. We also need to add a dependency from the ""lib"" target to the ""python"" target, because we can't link the Python libraries against the C++ library until it's built. That needs to be changed in sconsUtils. ",1 DM-2643,"Migrate Qserv to ssi v2","ssi v2 including comple objectification of the interface. Need to migrate qserv to the new interface.",6 DM-2646,"Switch to using shpgeom and remove duplicate code","Qserv is currently relying on a copy of the spherical geometry code (in core/modules/sg) instead of relying on the sphgeom module. This needs to be cleaned once we sort out the build issues with sphgeom (DM-2262).",1 DM-2647,"TOWG attendance ","remote operations discussion. ",1 DM-2648,"MGT for balance of April ","recruiting for open positions Work on accounting infrastrucutre. ""hardware"" contact - - outline to Jeff over the phone what is coming work through inventory infrastructure for materials for La Sereba review budget and effort projections. hear file system invesitigations meeting. ",3 DM-2649,"Histogram options","We have studied histogram options supported by Exoplanet Archive (http://exoplanetarchive.ipac.caltech.edu/cgi-bin/TblView/nph-tblView?app=ExoTbls&config=planets) We'd like to support similar options for our Histogram plot. The options are: - Column selection - Axes options: Linear, Log, Linear reversed, Log reversed Range selection (Auto, Manual) - Binning options Min, Max, Number of Bins (These can be assigned automatically: nBins = sqrt(nPoints)) ",12 DM-2652,"Prepare external http api for Firefly viewer for beta use",NULL,18 DM-2653,"Fix thread leak in Qserv","Qserv is currently leaking a thread per query. Executing a simple query list select count(*) from Object in a loop results in everything hanging after qserv is up to 67 threads.",6 DM-2655,"Prepare next gen Firefly JavaScript API Tools for beta",NULL,8 DM-2656,"Look into current transient alert event systems.","- Research how prior and current alert systems work, Skyalert in particular. - Install and try current working transient event alert system - Catalina. What can we learn from this? - Look at new technologies that might help - ZeroMQ messaging, message formats, forms of distribution and archiving, etc.",10 DM-2657,"Configuration mechanism for GalSim galaxy generation","This is an additional script for great3sims to allow simple configuration of the great3sims.run(). Most of the parameters which need to be set are in great3sims.constants.py, though some additional command line parameters may be needed for the run method.",2 DM-2658,"Build Psf Libraries from PhoSim Images","Takes output provided by Debbie from PhoSim runs and use them to create libraries of Psfs. Warp to remove camera distortion if necessary. This issue does not include figuring out what different categories of Psfs are required, but all of the process issues should be covered in this issue.",4 DM-2659,"Categorize Psfs and Distributions Required from PhoSim","Request a full focal plane of Psf images. Write code to allow them to be stored in a way which allows us to sample randomly from a full focal plane. There will be multiple such focal planes, so we also need to be able to pass the information to the measurement algorithm which will allow us to categorize measurements by visit. This will be done in the Psf Library building code, and will then be passes to the measurement algorithm through the great3sims code which constructs the data for the measurement algorithm.",2 DM-2660,"Produce HSC Psf sample for use in algorithm testing","Produce a set of well distributed Psfs from HSC data. As long as the Wcs info is also provided, the code to warp them should have been done in a separate issue.",4 DM-2661,"Alternative parameterized Psfs from PhoSim","Michael Schneider has suggested that he can do a better job of creating realistic Psfs from Psf models which he is working on, and which he intends to integrate into GalSim. These are intriguing, but depend on work which hasn't been done yet. When these models are fully available in GalSim and supported through the yaml configuration interface, we should work with them. But this is currently an ""as time permits"" issue.",6 DM-2662,"Prototype test harness for testing measurement algorithms","This is a relatively simple task, which will take the Galaxy images from the great3sims modifications and run measurement algorithms on the individual postage stamps. The result will be a catalog of the measurement outputs, cross-references against the galaxy and psf parmeters used for a given postage stamp. To do this, we need to combine information from the galaxy catalog and psf catalog into an input catalog for the algorithm. A source needs to be created for each galaxy which will contain at least the galaxy centroid and footprint relative to the postage stamp. The postage stamp with Psf appended and the above source much be fed to the measurement algorithm",4 DM-2663,"Do time tests running measurement algorithms against sample galaxies","Jim has suggest that we use cmodel to run these tests, since he is not committing to completing a complete shape measurement algorithm during the next sprint. So we will do our timing test using cmodel and shapelet approximation, and switch to the new algorithm from Jim when it is available.",5 DM-2664,"Find an adequate process platform for shape measurement tests","This issue requires an estimate of how many measurements we will need to run during S 15. And it also needs an estimate of how long it will take to measure a single galaxy. We should be able to guess how many galaxies are required to do an accurate assessment of a single parameterization of the shape measurement algorithm. We probably cannot accurately estimate how much of the parameter space of the shape measurement algorithm we will have to explore. The total amount of processing required should tell us whether this can be done with simple multi-core systems, or if a more sophisticated parallel process environment is required. There is a large additional task if ordinary multi-core processing isn't adequate, so this task may spawn a rather large additional issue.",4 DM-2665,"proof of concept types and providers for managing jenkins security settings","Proof of concept level implementation of native puppet types and providers for managing jenkins users, security realm, and authorization strategy.",20 DM-2666,"Create Analysis code for Constant Shear Tests","For any test of shear measurement vs. input shear (where input shear is constant), plot the measured shear vs. input shear and fit the multiplicative bias m and additive bias c.",8 DM-2667,"puppet types & providers for puppet security management","The current puppet-jenkins (and ansible, and chef) can not fully control jenkins users and security realm / authorization strategy settings. We should develop a proof of concept level limitation of naive puppet types and providers for users, security realm, and authorization strategy.",20 DM-2668,"Analyze bias vs. postage stamp size of galaxies","Vary the postage stamp size of simulated galaxies and access the effect of that size on the shear bias. This task will not require additional galaxy image generation, as the intent is to generate all the galaxies at a size which is liberally larger than the likely point where bias does not change with size. James Jee has indicated that 48 pixels on the LSST scale is not large enough enough for this bias to converge. It seems likely that we will need to generate our images at 64 or 96 pixels to get beyond this limit.",10 DM-2669,"resolve communication between JavaScript component and java server"," We are writing the web application client side code in JavaScript. JS interop will make it much easier to use GWT with JavaScript libraries. This task is to resolve the issues that may rise with this technology since it is new in GWT2.7. ",10 DM-2671,"Build 2015_05 Qserv Release","See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe.",1 DM-2672,"Build 2015_06 Qserv Release","See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe.",1 DM-2674,"Get meas_mosaic working on HSC data with LSST stack","We have an old, bitrotted version of meas_mosaic on the LSST side, created in a failed attempt to get it running on LSST PhoSim data. Now that we're making a serious effort to get HSC data running through the LSST pipeline, we'll need to get it running with the LSST pipeline at least on HSC data, which will probably involve just merging everything from the HSC side over, and then fixing it until it builds and runs. For this issue, we'll assume that we're going to use the Eigen backend for the matrix solver, rather than the MKL one we use on the HSC side. That will make it much slower (since MKL is multithreaded and we can't make Eigen multithreaded for just meas_mosaic), but hopefully still usable.",10 DM-2675,"Research Serf and Consul","Serf: https://serfdom.io Consul: https://www.consul.io ",4 DM-2678,"Investigate loading of binary data","The binary entries in qserv_testdata are stored as binary values in text files and there is no reason to believe that they are being read into the database correctly, see qserv_testdata/datasets/case01/data/Science_Ccd_Exposure.tsv.gz. Binary data from text files needs to be in hex format along with whatever other changes need to be made to reliably load the data into the database. Note that this story involves just investigating, it is not yet clear how much work will be needed to properly implement it. This story will help up understand the effort needed.",5 DM-2679,"Fix default LOAD DATA options","Integration tests in multi-node produced the following error during data loading: {code} 2015-05-01 17:03:03,030 - lsst.qserv.admin.dataLoader - CRITICAL - Failed to load data into non-partitioned table: Data truncated for column 'poly' at row 60 2015-05-01 17:03:03,031 - root - CRITICAL - Exception occured: Data truncated for column 'poly' at row 60 {code} The default options for MySQL LOAD DATA need to be fixed for this.",1 DM-2680,"v11.0 release",NULL,20 DM-2681,"Fix race condition in userQueryProxy","In UserQuery_kill, depending on timing, the call ""uqManager.get(session)->kill()"" can fail if kill is called more than once by user, because the session might get deleted by the earlier kill. To simulate this, I modified the code to delay the second kill as follows: {code} void UserQuery_kill(int session) { static int killNo = 0; killNo ++; LOGF_INFO(""EXECUTING UserQuery_kill(%1%), %2%"" % session % killNo); if (killNo > 1) { sleep(10); } uqManager.get(session)->kill(); } {code} We need to revisit if other functions in this class might suffer in similar way.",4 DM-2682,"Add missing empty-chunk-path on Ubuntu 14.04","QSERV_DATA_DIR/var/lib/qserv wasn't created on Ubuntu 14.04 and this was breaking loader script. It was working on SL7 for unknown reason. Creation of the directory has been added to qserv-czar config script.",1 DM-2683,"Fix case05 3009_countObjectInRegionWithZFlux freeze","This prevents 2014_05 release to pass integration tests.",1 DM-2684,"F17 Experiment with Non-Partitioned Tables","Test sharing of unpartitioned tables between worker nodes. This is something we claimed would work if we simply stuck them on a SAN, but never tested. Now is a good time to find out whether it actually works. If it fails, we need to re-think that part of the design. Shall we put the unpartitioned on a set of std replicated mysql nodes and attach them to the worker mysqld via the connect engine? Probably worth it to reconsider the overall architecture so that this can be integrated more elegantly (and maintainably) than just bolting on more moving parts. This epic involves: a) revisiting the numbers and checking if we could simply replicate non-partitioned tables on all nodes b) estimating realistic load on the non-partitioned tables c) playing with bringing the non-partitioned tables through mechanisms such as [connect engine|https://mariadb.com/kb/en/mariadb/connect-table-types-mysql-table-type-accessing-mysqlmariadb-tables/] If replicating on all nodes turns out to be too costly, we will arrange appropriate test bed (at NCSA?) and do the testing in S16 cycle. ",20 DM-2687,"Clean up FITS binary table writing","FITS binary table is being refactored by necessity on DM-2534, and while there's no similarly urgent need to clean up the writing code, we should do it at some point, as the refactoring of the read code broke some symmetries and made it even harder to follow the writing code that it already was.",4 DM-2689,"Review with Telefonica revised path Tololo - Pachon","A meeting in Santiago with Reuna and Telefonica to discuss the difference in price for the new path from Tololo to Pachon",2 DM-2691,"May 1 management","Met with A. Withers first cut read of scada plan. Contact session AURA -- explain gross contract changes accepted i principle)",1 DM-2692,"Rule for automatic replication in iRODS","Maintaining extra copies/replicas on separate resources is an important tenet in iRODS, with this practice considered key for prevention of data loss. The automatic replication of files upon ingest can be encoded via a system rule, so that data is preserved as a inherent part of storing in iRODS.",2 DM-2693,"Revisit mysql connections usage in integration tests","Recent changes in integration tests require too many connections. We need to understand what changes that is now requiring so many connections, and fix it.",4 DM-2694,"Revisit mysql connections from worker","Revisit the code that handles mysql connections in qserv. At the moment Qserv will maintain a connection per chunk-query, up to a hardcoded limit (GroupScheduler: 4, ScanScheduler:32). Also, we have to gracefully handle connection issues (such as dropped connection, or if we hit the max_connections limit).",8 DM-2695,"Prototype Ceph Deployment","Deploy Ceph on spare storage servers, with particular emphasis on deploying Ceph FS. This should likely take 8-15 story points over a period of about 3 weeks. This does not include a production deployment of Ceph for LSST. It is intended to help us gain insight into requirements for the initial production deployment. The story will primarily consist of effort from M. Elliott & W. Glick, with secondary effort from M. Freemon and B. Mather. ",20 DM-2696,"Research GPFS Server for Performant Access to Condo Storage","Work with NCSA SET to figure out requirements for LSST GPFS Server access to our Condo storage. Implementation will be a different story. Expect this to take 3-6 story points over the next couple of weeks.",1 DM-2698,"Fix connection leak","Fix connection leak: 1 connection is leaking per chunk-query, in practice ~30+ connections for a query that touches many chunks. It is a real blocker, and we need to fix it asap.",6 DM-2699,"Final cleanup of Query cancellation code","The query cancellation code that went in through DM-1716 works fine, however we feel it'd be good to do another pass and double check we are applying the cancellation consistently. Some potential places to clean: 1. in ccontrol/UserQuery.cc we changed the semantics of discard() 2. QueryRequest needs some cleanup: it'd be better to call Finished() from one place More regarding the former (from DM-1716 PR): ""if a query is cancelled, none of the cleanup below happens in discard() anymore -- presumably we are now waiting for object deletion to do the cleanup. If object deletion is sufficient to do this cleanup, do we need discard() at all anymore? It would be best if cleanup always occured in the same place rather than having two different control paths for it?"" Regarding the latter - see comment in https://jira.lsstcorp.org/browse/DM-1716",4 DM-2703,"Fix memory leak in Executive","There is a memory leak, most likely in Executive, related to _requesters. It looks like the ~MergingRequester() is never during normal operations (it is called when there are abnormal conditions and different parts of the code are triggered). As a result _infileMerger kept inside MergineRequester is not called either, which results in 2 connection leaks per query.",6 DM-2705,"Latin America Infinera rep to give presentation of equipment",NULL,4 DM-2707,"FY17 Research technologies potentially useful for Data Access",NULL,26 DM-2708,"Understand race condition in Executive::_dispatchQuery","Inserting a log (presumably just a delay) in Executive::_dispatchQuery after the new QueryResource but before the Provision call causes queries to fail. The particular test query was ""select count(*) from Object"" on test case 01.",2 DM-2709,"Convert the ds9 interface to follow RFC-42","RFC-42 (provide a backend-agnostic interface to displays) being accepted, please implement it. For now, provide compatibility code so that the old way (import lsst.afw.display.ds9) still works. ",6 DM-2710,"Mutex use before creation","qana/QueryPlugin.cc contains a static boost::mutex, that is used by static class member functions to register plugin implementations. Its constructor is not guaranteed to be called before the static registerXXXPlugin (see e.g. qana/AggregatePlugin.cc) instances use it to register plugin classes.",1 DM-2711,"Migrate boost:thread to std::thread","We are mixing boost and std threading libraries. This should be cleaned up - use std:thread consistently everywhere.",5 DM-2712,"Migrate boost::shared_ptr to std::shared_ptr","We are mixing boost and std shared_ptrs. This should be cleaned up - use std:shared_ptr consistently everywhere. In a few places we have other types of pointers, (e.g weak_ptr). Migrate these too.",2 DM-2715,"Add missing includes unistd.h for gcc 4.9.2",NULL,1 DM-2716,"Fix connection leak (2nd iteration)","Fix connection leak (and memory leak and thread leak) -- we are leaking 2 per query.",2 DM-2717,"Add test involving many chunks","It might be useful to add a test to the integration test suite that involves a large number of chunks per node. I think I'd try something like 200-300 chunks. I'd 1. add case06 2. get one table, say Object from case05 and configure partitioning to ensure we have 200-300 chunks. 3. Run several queries that touch all chunks.",5 DM-2718,"Upgrade EUPS used by lsstsw","As discussed, bump it up when you get a chance please. ",1 DM-2720,"Migrate boost::scoped_ptr to std","We have a few places where we are using boost::scoped_ptr. Given we migrated shared_ptrs, we might want to move scoped_ptrs too (most likely to std::unique_ptr).",1 DM-2722,"Revisit design of query poisoner","As we discovered through DM-2698, poisoner tends to hold onto query resources even after the query completes. We should revisit whether than can be redesigned and improved, so that when query finishes, all resources related to that query are immediately automatically released. This story involves just the planning part, implementation will be done through separate stories.",1 DM-2723,"LOE - Week ending 5/8/15",NULL,12 DM-2724,"LOE - Week ending 5/15/15",NULL,3 DM-2725,"LOE - Week ending 5/22/15",NULL,3 DM-2726,"LOE - Week ending 5/29/15",NULL,12 DM-2727,"Package Python requests package","To complete DM-2593 we need to package and install `requests` as a separate package instead using one from anaconda.",1 DM-2728,"Build should fail if node.js is not present","Problem: I built Firefly by mistake w/o having node on my path. The build didn't signal any errors, but generated an unusable webapp that wouldn't load. Expected behavior: the build should have failed and warned the user that node.js is missing.",2 DM-2729,"Fix a few more g++ 4.9.2 compatos","Some of the recent boost -> std changes don't compile/link under gcc 4.9.2, because of some poor #include hygiene (including when we should include , not explicitly including , etc.) Also, -pthread linker option is required when using std::thread under gcc 4.9.2. ",1 DM-2733,"Generalize / Simplify Facade ","Daniel started thinking about simplifying Facade, here is some unfinished code from him {code} /// Unfinished. Planned to be a re-thinking of Facade that collapses some /// genericity and simplifies things using the assumption of running on a /// snapshot. class FacadeSnapshot : public Facade { public: StringMap _map; // Path --> key FacadeSnapshot() { } virtual bool containsDb(std::string const& dbName) const { if (dbName.empty()) { LOGF_DEBUG(""Empty database name passed.""); throw NoSuchDb(""""); } string p = _prefix + ""/DBS/"" + dbName; bool ret = (_map.find(p) != _map.end()); LOGF_DEBUG(""containsDb(%1%): %2%"" % dbName % ret); return ret; } virtual bool containsTable(std::string const& dbName, std::string const& tableName) const { if (!containsDb(dbName)) { throw NoSuchDb(dbName); } if (tableName.empty()) { LOGF_DEBUG(""Empty table name passed.""); throw NoSuchTable(""""); } string p = _prefix + ""/DBS/"" + dbName + ""/TABLES/"" + tableName; bool ret = (_map.find(p) != _map.end()); LOGF_DEBUG(""containsTable returns: %1%"" % ret); return ret; } virtual bool tableIsChunked(std::string const& dbName, std::string const& tableName) const { if (!containsTable(dbName, tableName)) { throw NoSuchTable(dbName + ""."" + tableName); } string p = _prefix + ""/DBS/"" + dbName + ""/TABLES/"" + tableName + ""/partitioning""; bool ret = (_map.find(p) != _map.end()); LOGF_DEBUG(""%1%.%2% %3% chunked."" % dbName % tableName % (ret?""is"":""is NOT"")); return ret; } virtual bool tableIsSubChunked(std::string const& dbName, std::string const& tableName) const { string p = _prefix + ""/DBS/"" + dbName + ""/TABLES/"" + tableName + ""/partitioning/"" + ""subChunks""; StringMap::const_iterator m = _map.find(p); bool ret = (m != _map.end()) && (m->second == ""1""); LOGF_DEBUG(""%1%.%2% %3% subChunked."" % dbName % tableName % (ret ? ""is"" : ""is NOT"")); return ret; } virtual bool isMatchTable(std::string const& dbName, std::string const& tableName) const { LOGF_DEBUG(""isMatchTable(%1%.%2%)"" % dbName % tableName); if (!containsTable(dbName, tableName)) { throw NoSuchTable(dbName + ""."" + tableName); } string p = _prefix + ""/DBS/"" + dbName + ""/TABLES/"" + tableName + ""/match""; StringMap::const_iterator m = _map.find(p); bool ret = (m != _map.end()) && (m->second == ""1""); LOGF_DEBUG(""%1%.%2% is %3% a match table"" % dbName % tableName % (ret ? """" : ""not "")); return ret; } #if 0 virtual std::vector getAllowedDbs() const { }; virtual std::vector getChunkedTables(std::string const& dbName) const; virtual std::vector getSubChunkedTables(std::string const& dbName) const; virtual std::vector getPartitionCols(std::string const& dbName, std::string const& tableName) const; virtual int getChunkLevel(std::string const& dbName, std::string const& tableName) const; virtual std::string getDirTable(std::string const& dbName, std::string const& tableName) const; virtual std::string getDirColName(std::string const& dbName, std::string const& tableName) const; virtual std::vector getSecIndexColNames(std::string const& dbName, std::string const& tableName) const; virtual StripingParams getDbStriping(std::string const& dbName) const; virtual double getOverlap(std::string const& dbName) const; virtual MatchTableParams getMatchTableParams(std::string const& dbName, std::string const& tableName) const; private: #endif }; {code}",14 DM-2734,"Add config file for test dataset 04 tables","Following the changes to default LOAD DATA settings in DM-2679, two tables in test case 04 need to have a config file to include their in.csv format.",1 DM-2735,"optimistic matcher may match the same reference object to more than one source","The optimistic pattern matcher in meas_astrom, adapted from hscAstrom, does not check if reference objects have been used before when finding the reference object nearest to each source. As a result the same reference object may be matched to more than one source. This should not happen.",4 DM-2736,"Log xrootd client debug messages in Qserv czar","xrootd client print it's debug messages to stdout. This ticket aims at redirecting them to Qserv logger, if possible.",4 DM-2737,"Build a DiscreteSkyMap that covers a collection of input exposures","This is essentially a rehash of the old trac Ticket #[2702| https://dev.lsstcorp.org/trac/ticket/2702], originally reported by [~jbosch], which reads: ""I'd like to add a Task and bin script to create a DiscreteSkyMap that bounds a set of calexps specified by their data IDs. This makeDiscreteSkyMap.py could be used instead of makeSkyMap.py when the user would rather compute the pointing and size of the skymap from the input data than decide it manually."" The work was done by [~jbosch] & [~price] and exists on branch {{u/price/2702}} in {{pipe_tasks}}, but it was never merged to master. I plan to simply rebase the commits in that branch onto master.",1 DM-2738,"Remove #include ""XrdOuc/XrdOucTrace.hh"" from Qserv code","See next emails: Hi Fabrice, Absolutely! Andy On Wed, 13 May 2015, Fabrice Jammes wrote: > Hi Andy, > > Thanks, > > In my understanding, you're ok if I remove the existing > #include ""XrdOuc/XrdOucTrace.hh"" > from Qserv source code. I'll do it soon. > > Have a nice day, > > Fabrice > > Le 12/05/2015 23:41, Andrew Hanushevsky a écrit : >> Hi Fabrice, >> >> Well, no. We have a long-standing approach that qserv should not depend on anything outside of XrdSsi public interfaces. This is the only way to easily protect sqserv code from infrastructure changes. So, I would not. If you want to copy something like that for >> >> qserv please do, it's simple enough. But in the end qserv needs to be self-contained in that it does not depend on xrootd code just the public ssi interfaces. >> >> Andy >> >> -----Original Message----- From: Fabrice Jammes >> Sent: Tuesday, May 12, 2015 9:06 AM >> To: Andrew Hanushevsky >> Subject: About xrdssi client logging >> >> Hi Andy, >> >> Hope you're doing well. >> Could you please tell me if its usefull to include >> #include ""XrdOuc/XrdOucTrace.hh"" >> in our xrdssi client code? >> >> Indeed client seems to only print DBG macro output, that's why I was >> wondering if XrdOucTrace was only use on the server side. >> If yes, I will remove it from our client. >> >> Thanks, and have a nice day, >> >> Fabrice ",1 DM-2740,"Make ANetAstrometryTask more configurable","The current ANetAstrometryTask has a solver that is not easy to retarget. This makes testing with hscAstrom needlessly difficult. My suggestion is to make the solver a true Task instead of a task-like object, and make it retargetable using a ConfigurableField instead of a ConfigField. This is very easy to do because the solver is already a task in all but name. ",2 DM-2743,"sandbox selection of newinstall.sh source url","Frossie would like the ability to control the source URL for the newinstall.sh script in sandbox-stackbuild. The newinstall.sh installation logic needs to be migration to the puppet-lsststack module, converted into a defined type, and have unit+ acceptance tests written for it.",1 DM-2744,"Second Review with Chris Smith AURA head","Went over the process relating to AURA and NSF",4 DM-2745,"Design of the summit network computer facility","jeff ",40 DM-2748,"Add clear message when integration test fails","Integration test fails without printing a clear message at the end, and for now a query is broken: 0011_selectDeepCoadd.txt but it isn't printed at the end of tet output.",2 DM-2750,"Fix case04/0011_selectDeepCoadd.txt","It seems --config=/path/to/table.cfg param can be duplicated (see dbLoader l.77 and QservDbLoader l. 87) Futthermore there is an enclosing pb and it can be solved for this query by passing correct cfg table (which in.csv.enclose correct parameter), but then next query fails: it seems some cfg parameters of table.cfg aren't managed correctly by the loader in plain MySQL mode. This need further investigations.",6 DM-2751,"Allow lsst/log library to log PID on the C++ side","lsst/log should be able to log application PID",3 DM-2752,"db 10.1+4 tests randomly fail with python egg installation error","The unit tests for DB seem to fail at random and always pass on a second build attempt. My hunch is that multiple tests are running in parallel all attempting to install the mysql module but I haven't investigated. {code} db: 10.1+4 ERROR (0 sec). *** error building product db. *** exit code = 2 *** log is in /home/build0/lsstsw/build/db/_build.log *** last few lines: ::::: [2015-05-15T19:12:35.557258Z] scons: done reading SConscript files. ::::: [2015-05-15T19:12:35.558276Z] scons: Building targets ... ::::: [2015-05-15T19:12:35.558409Z] scons: Nothing to be done for `python'. ::::: [2015-05-15T19:12:35.570007Z] makeVersionModule([""python/lsst/db/version.py""], []) ::::: [2015-05-15T19:12:35.686733Z] running tests/testDbLocal.py... running tests/testDbRemote.py... running tests/testDbPool.py... failed ::::: [2015-05-15T19:12:35.695011Z] passed ::::: [2015-05-15T19:12:35.698811Z] passed ::::: [2015-05-15T19:12:35.706360Z] 1 tests failed ::::: [2015-05-15T19:12:35.706703Z] scons: *** [checkTestStatus] Error 1 ::::: [2015-05-15T19:12:35.708443Z] scons: building terminated because of errors. {code} {code} [root@ip-192-168-123-151 .tests]# cat * tests/testDbLocal.py Traceback (most recent call last): File ""tests/testDbLocal.py"", line 53, in from lsst.db.db import Db, DbException File ""/home/build0/lsstsw/build/db/python/lsst/db/db.py"", line 49, in import MySQLdb File ""build/bdist.linux-x86_64/egg/MySQLdb/__init__.py"", line 19, in File ""build/bdist.linux-x86_64/egg/_mysql.py"", line 7, in File ""build/bdist.linux-x86_64/egg/_mysql.py"", line 4, in __bootstrap__ File ""/home/build0/lsstsw/anaconda/lib/python2.7/site-packages/setuptools-5.8-py2.7.egg/pkg_resources.py"", line 937, in resource_filename File ""/home/build0/lsstsw/anaconda/lib/python2.7/site-packages/setuptools-5.8-py2.7.egg/pkg_resources.py"", line 1632, in get_resource_filename File ""/home/build0/lsstsw/anaconda/lib/python2.7/site-packages/setuptools-5.8-py2.7.egg/pkg_resources.py"", line 1662, in _extract_resource File ""/home/build0/lsstsw/anaconda/lib/python2.7/site-packages/setuptools-5.8-py2.7.egg/pkg_resources.py"", line 1003, in get_cache_path File ""/home/build0/lsstsw/anaconda/lib/python2.7/site-packages/setuptools-5.8-py2.7.egg/pkg_resources.py"", line 983, in extraction_error pkg_resources.ExtractionError: Can't extract file(s) to egg cache The following error occurred while trying to extract file(s) to the Python egg cache: [Errno 17] File exists: '/home/build0/.python-eggs' The Python egg cache directory is currently set to: /home/build0/.python-eggs Perhaps your account does not have write access to this directory? You can change the cache directory by setting the PYTHON_EGG_CACHE environment variable to point to an accessible directory. tests/testDbPool.py /home/build0/lsstsw/anaconda/lib/python2.7/site-packages/setuptools-5.8-py2.7.egg/pkg_resources.py:1032: UserWarning: /home/build0/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable). 05/15/2015 07:12:35 root WARNING: Required file with credentials '/home/build0/.lsst/dbAuth-test.txt' not found. tests/testDbRemote.py /home/build0/lsstsw/anaconda/lib/python2.7/site-packages/setuptools-5.8-py2.7.egg/pkg_resources.py:1032: UserWarning: /home/build0/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable). 05/15/2015 07:12:35 root WARNING: Required file with credentials '/home/build0/.lsst/dbAuth-testRemote.txt' not found. {code}",1 DM-2754,"Write example-based documentation for multiband processing","The multi-band coadd processing tasks we're porting over from the HSC side don't have the high-quality example-based documentation we typically provide for Tasks on the LSST side, so we need to write it from scratch.",6 DM-2755,"Improve selection criteria for sources","Dominique Boutigny has demonstrated that one reason the new astrometry task is working so poorly is that it is not selective enough about which sources it uses. This ticket is to be used to improve that situation. Another problem Dominique discovered is that the TAN-SIP WCS fitter needs to be iterated to work properly, and that work may also be done on this ticket. ",4 DM-2756,"Configure NCSA LSST Perfsonar Host",NULL,3 DM-2757,"Administrative - 6-2015","Meetings and reporting and such",2 DM-2758,"Create LSST wiki documentation for LHN effort",NULL,3 DM-2759,"Onboarding Humberto","Efforts in helping new employee Humberto come up to speed in his role as lead on perfsonar deployments",1 DM-2762,"Avoid leaking memory allocated by mysql_thread_init","mysql/MySqlConnection.cc contains the following comment: {code} // Dangerous to use mysql_thread_end(), because caller may belong to a // different thread other than the one that called mysql_init(). Suggest // using thread-local-storage to track users of mysql_init(), and to call // mysql_thread_end() appropriately. Not an easy thing to do right now, and // shouldn't be a big deal because we thread-pool anyway. {code} The comment is not really correct with regards to thread pooling. Instead, each rproc::InfileMerger has an rproc::InfileMerger::Mgr which contains a util::WorkQueue that spawns a thread, and so we are failing to call mysql_thread_end at least once per user query. This has been verified using the memcheck valgrind tool. ",3 DM-2763,"Mountain - Base fiber implementation","Aquire, install, and test fiber connecting Mountain - Base. The fiber will follow a path along roads from Cerro Pachon to Cerro Tololo and down to the AURA gate, where it will connect with Telefonica fiber bundle to La Serena.",40 DM-2764,"Improve management of ColSchema.hasDefault and ColSchema.defaultValue","Managing default values in protobuf and result table isn't optimized for now. Indeed all values are packed in protobuf, whereas default values could be removed from protobuf messages (in .QueryAction::Impl::_fillRows()) It is interesting to monitor performance when packing default values, and when not, and then improve the code related to default value management, or completely remove it.",10 DM-2765,"AURA Traffic Utilizing the Fiber Link at 10Gbps","Obtain equipment in order to light the fibers up to 10Gbps and run live traffic for Tololo and Pachon over the link",30 DM-2766,"Fix ORDER BY in integration test query case03 0019.1.0 ","ORDER BY fails sometimes for unknown reason, see datasets/case03/queries/0019.1.0_selectRunDeepSourceDeepcoaddDeepsrcmatchRefobject.sql.FIXME",4 DM-2768,"investigate decomposition of stack build into independent packages","In order to obtain per package build time, test time, coverage, or virtually any per component metric ,the CI build needs to be decomposed from a single large integrated build into per package jobs with an overall work flow representing the dependency graph. This is also needed for binary artifacts to be passed between builds step and/or binary packages.",10 DM-2769,"sconsUtil has a hard dependency on EUPS for both tests and installation","After some discussion on Data Management, its clear that sconsUtils is a hard requirement on EUPS for both tests and installation. It was decided by RFC-44 that tests should not depend on EUPS. However, I'd argue that sconsUtils should also not depend on EUPS as any package that uses sconsUtils (the virtual entirety of the stack) can not build or run tests without the presence of EUPS. The current situation is that the complete stack has a hard dependency on EUPS. Attempting to build sconUtils without the presence of EUPS. The tests fail attempting to import the eups module. {code} $ SCONSUTILS_DIR=. scons -Q Unable to import eups; guessing flavor CC is gcc version 4.8.3 Checking for C++11 support C++11 supported with '-std=c++11' Unable to import eups; guessing flavor Doxygen is not setup; skipping documentation build. ImportError: No module named eups: File ""/home/vagrant/sconsUtils/SConstruct"", line 9: scripts.BasicSConstruct.initialize(packageName=""sconsUtils"") File ""python/lsst/sconsUtils/scripts.py"", line 106: SCons.Script.SConscript(os.path.join(root, ""SConscript"")) File ""/usr/lib/scons/SCons/Script/SConscript.py"", line 609: return method(*args, **kw) File ""/usr/lib/scons/SCons/Script/SConscript.py"", line 546: return _SConscript(self.fs, *files, **subst_kw) File ""/usr/lib/scons/SCons/Script/SConscript.py"", line 260: exec _file_ in call_stack[-1].globals File ""/home/vagrant/sconsUtils/tests/SConscript"", line 5: import eups {code} Attempting to bypass the test failures: {code} [vagrant@jenkins-el7-1 sconsUtils]$ rm -rf tests [vagrant@jenkins-el7-1 sconsUtils]$ SCONSUTILS_DIR=. scons -Q install Unable to import eups; guessing flavor CC is gcc version 4.8.3 Checking for C++11 support C++11 supported with '-std=c++11' Error with git version: uncommitted changes Found problem with version number; update or specify force=True to proceed {code}",6 DM-2770,"sconsUtil install target does not respond to either force=True or --force","I've been unable to figure out how to bypass the install 'force' check, but have confirmed that this is the correct expression by commenting it out: https://github.com/lsst/sconsUtils/blob/54c983ffe9714a33657c4388de3506fe7a40518d/python/lsst/sconsUtils/installation.py#L92 {code} $ SCONSUTILS_DIR=. scons -Q force=True install Unable to import eups; guessing flavor CC is gcc version 4.8.3 Checking for C++11 support C++11 supported with '-std=c++11' Error with git version: uncommitted changes Found problem with version number; update or specify force=True to proceed {code} ",1 DM-2775,"Improve SIP fitting","Dominique Boutigny says ""the tan-sip fitter is very sensitive to bad matches. This is a weakness of the fitter and I think that it could (should) be rewritten in such a way to reject the outliers internally."" This has resulted in iteration in the matching (DM-2755), which should be unnecessary (or at least minimised). Additionally, it seems the SIP fitter fits for x and y in subsequent iterations, which can confuse users. We should: 1. Make the SIP fitter fit for x and y concurrently. 2. Add rejection iterations in the SIP fitter. 3. Remove or minimise the iterations in the matching.",6 DM-2777,"Fix races in BlendScheduler","_integrityHelper() from wsched/BlendScheduler inspects a map of tasks and is sometimes called without holding the corresponding mutex. My theory is that it is observing the map in an inconsistent state, leading to assert failure and hence worker death, and finally to hangs/timeouts on the czar.",2 DM-2778,"HSC backport: allow for use of Approximate model in background estimation","This issue involves transferring changesets from the following HSC issues: - [HSC-145|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-145] Investigate approximating rather than interpolating backgrounds - [HSC-1047|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1047] Background object cannot be loaded with butler - [HSC-1213|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1213] Set background 'approximate' control settings when background control is created. - [HSC-1221|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1221] tests failing in ip_diffim - [HSC-1217|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1217] Verify backgroundList IO works properly when Approximate is enabled in background control - HSC JIRA The Approximate (Chebyshev) approach greatly improves the background subtraction around bright objects compared with the interpolation scheme currently in use (which over-subtracts near bright objects).",6 DM-2779,"Fix race in Foreman","The Foreman implementation passes a TaskQueue pointer corresponding to running tasks down to the task scheduler without holding a lock. This means that the scheduler can inspect the running task list (usually to determine its size) while it is being mutated.",2 DM-2780,"Document and test how to log PID via lsst/log",NULL,2 DM-2781,"push PID in lsst/log MDC in a C++ plugin (for xrootd)",NULL,2 DM-2782,"Firefly Tools API: Add advance region support","Firefly Tools API: Add advance region support Improve firefly's region functionality to support a ""dynamic region"". Data can be added or removed from this region by API calls. Allow any amount of region lines to be added or removed. Make sure performance is good. Also, document the current Firefly region support.",2 DM-2783,"Control firefly viewer tri-view mode","When table data is add to Firefly Viewer, control whether it goes into tri-view or just overlay data on FITS, or just does an XYPlot, etc",6 DM-2784,"Add Firelfy Tools API controlled Pan and Zoom",NULL,2 DM-2785,"FFTools python wrapper: make launch Browser smarter. ","FireflyClient.launchBrowser() needs to send an event to the server who will attempt to guess if there is an existing connection. It will not be launch in that case. This way it can be called every time without creating tons of tabs, who are all talking to the same channel. Also, launchBrowser really should not return until the tab is ready to receive events from the websocket channel. Both of these feature are going to take some thought on how to do. This is a multi-threaded problem on both the client and the server. ",4 DM-2786,"FFTools api, wrapper: upload region file from memory like fits file",NULL,1 DM-2787,"Footprint dilation performance regression","In DM-1128 we implemented span-based dilation for footprints. A brief test on synthetic data indicated that this was a performance win over the previous version of the code. In May 2015, this code was merged to HSC and applied to significant quantities of real data for the first time. A major performance regression was identified: {quote} [May-9 00:26] Paul Price: processCcd is now crazy slow. [May-9 00:29] Paul Price: Profiling... [May-9 00:40] Paul Price: I'm thinking it's the Footprint grow code... [May-9 00:44] Paul Price: And the winner is…. Footprint construction: [May-9 00:44] Paul Price: 2 0.000 0.000 702.280 351.140 /home/astro/hsc/products/Linux64/meas_algorithms/HSC-3.8.0/python/lsst/meas/algorithms/detection.py:191(makeSourceCatalog)
 2 0.005 0.002 702.274 351.137 /home/astro/hsc/products/Linux64/meas_algorithms/HSC-3.8.0/python/lsst/meas/algorithms/detection.py:228(detectFootprints)
 15 0.001 0.000 698.597 46.573 /home/pprice/hsc/afw/python/lsst/afw/detection/detectionLib.py:3448(__init__)
 15 698.596 46.573 698.596 46.573 {_detectionLib.new_FootprintSet} [May-9 00:53] Paul Price: If I revert HSC-1243 (""Port better Footprint-grow code from LSST""), then the performance regression goes away. @jbosch @jds may be interested... {quote} The source of the regression must be identified and resolved for both HSC and LSST.",5 DM-2789,"rename CameraMapper.getEupsProductName() to getPackageName() and convert to abstract method","Per discussion on this PR related to DM-2636: https://github.com/lsst/daf_butlerUtils/pull/1#issuecomment-104785055 The CameraMapper.getEupsProductName() should be renamed to getPackageName() and converted to an abstract method. This will eliminates a runtime, and thus ""test time"", dependency on EUPS. As part of the rename/conversion, all subclasses that are not already overriding getEupsProductName() will concurrently need to have getPackageName() implemented.",3 DM-2790,"meas_modelfit not in full-stack doxygen build","I'm fairly certain meas_modelfit is included in lsst_apps and hence in CI, but id doesn't seem to be included in the LSST Doxygen build: http://lsst-web.ncsa.illinois.edu/doxygen/x_masterDoxyDoc/search.php?query=modelfit",1 DM-2792,"Make the new astrometry task the default task","The new astrometry task should be the default astrometry task, but we need to make sure it is good enough first.",1 DM-2793,"Improve behavior of new matcher on highly distorted fields","The optimistic matcher used by the new astrometry task probably does not handle highly distorted fields well. The issue is that it tries to match in X-Y space, and if that has significant curvature then the match is not optimal. I suggest matching in RA/Dec space, as per Tabur's original algorithm (on which this code is based). This is simple and easy to understand An alternative is to use the old technique of ""undistorting"" source and reference object positions before matching. This works, but is complicated, difficult to understand and adds an unnecessary step. ",8 DM-2794,"Fiber from Tololo to Pachon","Preparation for fiber cable from Cerro Tololo to Cerro Pachon",5 DM-2795,"Various fixes for broken code within display=True clauses and/or using --debug","Running with the display and/or debug options turned on is revealing many instances of code that is now suffering from bit rot. This ticket will be used to track those encountered while trying to debug issues arising while porting HSC code and running processing tasks on real data.",4 DM-2797,"Implement HSC improvements to Colorterm","Paul Price recommended some HSC changes for the Colorterm class. To quote Paul: changed Colorterms so it's not a global, and it can now be configured using Config. See https://github.com/HyperSuprime-Cam/meas_astrom/blob/master/python/lsst/meas/photocal/colorterms.py This sounds useful. Note that the HSC colorterms.py is in meas_astrom but as of DM-1578 the LSST version is in pipe_tasks.",6 DM-2799,"Tests for daf_butlerUtils should not depend on obs_lsstSim","Currently two of the tests in {{daf_butlerUtils}} depend on {{obs_lsstSim}}. They will never run in a normal build because {{obs_}} packages can not be a dependency on {{daf_butlerUtils}}. After discussing the options with [~ktl] the feeling is that {{ticket1640}} should be rewritten to remove the dependency and {{ticket1580}} can probably be removed.",2 DM-2800,"Review with Contractors preparing fiber path","Hold conversations with the two major fiber laying contractors to prepare the path from Tololo to Pachon with a trench",2 DM-2801,"Document NCSA Wide Area Network status now and in the near future","Write up a document explaining how the wide area network is evolving at NCSA.",2 DM-2802,"W16 Finish Implementing Database & Table Mgmt","Implement table and database deletion.",29 DM-2803,"Adapt multi-node tests to latest version of qserv / loader","The multi-node integration tests have to be updated to work with the latest changes to qserv, in particular the loader, which broke already working tests lately.",8 DM-2804,"Implement query metadata skeleton","Skeleton implementation of the Query Metadata - including the APIs and core functionality (accepting long running query and saving the info about it)",8 DM-2805,"Complete Query Metadata Implementation","Including query abort",10 DM-2806,"Run large scale tests",NULL,6 DM-2807,"Chile National Links Contracts Negotiation","The negotiations and conversations that have occurred with firstly Entel and finally Reuna and Telefonica for dark fiber between La Serena and Santiago",60 DM-2808,"La Serena - Santiago Dark Fiber Acquisition","Acquiring the fiber between La Serena and Santiago and testing the segments. This will be carried out by Reuna with some collaboration with LSST personnel",15 DM-2809,"La Serena - Santiago Link Equipment installation","Reuna will install the amplifiers and the DWDM equipment for the link between La Serena and Santiago with collaboration with LSST personnel ",20 DM-2810,"La Serena - Santiago LInk with Live Traffic flowing","The link is established and tested with Live LSST data running over the 100G Lambda",10 DM-2813,"Mountain to Base Implementation Plan Feasibility Check/Reevaulation","The Installation of the fiber on the AURA property from gate to Pachon via Tololo. Testing that portion of the link. Connecting the 4 dark fibers supplied by Telefonica and testing end to end.",80 DM-2814,"Fiber lay between Gate to Tololo","Telefonica will install the fiber cable from the Gatehouse to Tololo. This will be tested once terminated. We will oversee and monitor the installation and be present for testing.",40 DM-2815,"Fiber Install from Tololo to Pachon ","Telefonica will install the cable from Tololo to Pachon Sahred Infrastructure building and terminate. Testing of this portion will be carried out.",20 DM-2816,"Connection of Telefonica 4 Fibers at AURA Gate ","The connection will made with the Telefonica 4 dark fibres to the 24 fiber cable from the gate to Pachon. Tests will then take place over the whole mountain to base link.",10 DM-2817,"Creation of chunked views in wmgr","Current implementation of the creating chunks for views in wmgr is likely not doing right thing. Need to find an example of the partitioned views and implement correct procedure.",3 DM-2818,"Document architecture of the data loader","Fabrice requested documentation for the overall architecture of the data loader.",2 DM-2819,"Install 10Gbs Transceivers at the ends of the fibers and test","Purchase 2x10Gbs Transceivers and install at Pachon and La Serena. Run tests to confirm integrity of the link Utilize the link for AURA live traffic on the fiber backbone",20 DM-2820,"AURA traffic utilizing 100Gbs Lambda","AURA acquires their DWDM end nodes and installs. Tests and live traffic flows.",30 DM-2821,"Contract Negotiations for Chilean links","Defining the contracts between AURA and Reuna and Reuna and Telefonica who is supplying and installing the fibers for the La Serena to Santiago link",20 DM-2822,"Mountain - Base Contract and Execution","Defining the contracts and execution between AURA and Reuna and Reuna and Telefonica who is supplying and installing the fibers for the mountain to base link",30 DM-2823,"La Serena - Santiago Early Diverse Path","Reuna will provide a 4Gbps path via the legacy fiber path to Santiago",10 DM-2824,"La Serena - Santiago Diverse Path Final Capacity","Reuna will upgrade the capacity from 4Gbps to a minimum of 40Gbps over the legacy fiber route to Santiago",10 DM-2825,"La Serena - Santiago Fiber Tests over 3T cable","The 3T cable from La Serena to Santiago is expected to be completed September 2015 at which time Reuna can test the segments along the route. ",20 DM-2826,"Mountain - Base AURA link upgrade to 100Gbps","AURA will obtain their DWDM equipment end nodes and install on the backbone from the summits to La Serena.",30 DM-2827,"Implement RESTful interfaces for Database (POST)","Implement RESTful interfaces for Database (see all D* in https://confluence.lsstcorp.org/display/DM/API), based on the first prototype developed through DM-1695. The work includes adding support for returning appropriately formatted results (support the most common formats). This covers ""POST"" type requests only, ""GET"" will be handled separately.",8 DM-2830,"orphan threads in archive DMCS","The archive DMCS can experience orphaned threads if a connection is made from external processes waiting for data to arrive. If the external process goes away, the thread that was created to handle that connection will be waiting on a data structure to be updated. If the data doesn't arrive, the thread remains alive when it should be checking to see if the connection that created it is still viable, and die if it isn't viable.",17 DM-2833,"LOE - Week ending 6/5/15",NULL,5 DM-2834,"LOE - Week ending 6/12/15",NULL,3 DM-2835,"LOE - Week ending 6/19/15",NULL,3 DM-2836,"LOE - Week ending 6/26/15",NULL,1 DM-2837,"Add unit tests for the new colorterms code","The new colorterms code that we adopted from HSC may not have complete unit tests. The existing colorterms test is pretty good, but may have holes. I'm more concerned about the unit test for PhotoCalTask, which does not apply colorterms at all (likely an existing issue). Also, be sure to test that the obs_cfht config override loads correctly (presumably with a unit test in obs_cfht) and similarly for obs_subaru.",4 DM-2838,"Documentation and testing for Firefly Javascript and Python API","Document, polish and test Firefly Javascript and Python APIs - Proofread and polished all the documentation, added missing docs - Tested all the examples and API methods. Updated test cases as needed.",10 DM-2839,"sconsUtils should notice when SWIG python file has been modified","Currently {{scons}} will not rerun tests if a {{.i}} file has been modified if the only outcome of that modification was a change to the {{.py}} file. {{sconsUtils}} should be modified to look for changes in both SWIG output files.",2 DM-2840,"Add support for listing async queries","Modify mysql proxy and implement ""show processlist"" command, which should display list of currently running queries.",4 DM-2841,"Write Qserv User Guide","It'd be useful to write a document about Qserv geared towards users, describing what queries Qserv supports now, what will be supported in the future, what restrictions we are imposing and such. ",8 DM-2843,"Distributed Hash Table prototyping",NULL,20 DM-2844,reserved,NULL,10 DM-2845,"Near Neighbor Optimizations","To optimize near neighbor queries we are maintaining overlap tables and subchunking. It'd be useful to revisit that. Getting rid of subchunks would simplify qserv code. This epic involves * testing speed of in-database near neighbor queries without subchunking, including how sensitive the optimizer is for these types of queries * exploring possibility of precalculating and storing near neighbors, perhaps per subchunk ",15 DM-2847,"SUI Firefly server side Python job management","In order to support Camera team needs and L3 data production, Firefly server needs to be able to start a Python job with proper input data and get the output data as a result of running the Python job. This will make the future integration of Firefly and DM pipeline stack much easier. ",40 DM-2849,"Tweaks to OO display interface","When I wrote the initial version of display_firefly I found a few minor issues in the way I'd designed the Display class; at the same time, [~lauren] found some missing functions in the backward-compatibility support for ds9. Please fix these; note that this implies changes to afw, display_ds9, and display_firefly. ",2 DM-2850,"getSchemaCatalogs() breaks Task encapsulation","The {{getSchemaCatalogs()}} method was added to {{Task}} to allow {{CmdLineTasks}} to introspect their subtasks for schemas they produce, but it requires the subtasks to report the schemas by butler dataset. This limits subtask reusability by locking them into producing a particular Butler dataset (or, as in DM-2191, requiring additional arguments from their parent task that they wouldn't need with a better design). Instead, we should have per-subtask-slot interfaces (i.e. an interface for all subtasks that could fill a particular role in a CmdLineTask) for how the parent tasks should retrieve their schemas. This will require `CmdLineTask` subclasses to implement the `writeSchemas` method themselves, instead of inheriting an implementation from `CmdLineTask` itself.",2 DM-2851,"Build the recent 10.1 release & Gather strace logs for file system testing","We build the recent Version 10.1 stack release in a Centos 6.6 docker container. As we do so, we also gather strace logs for candidate packages (for example, afw) for analysis within an effort to create load simulators for file system testing/profiling. As another product of the effort, I will make a docker image of the latest release installed on Centos 6.6 and push to docker hub.",4 DM-2852,"AAA requirements document",NULL,2 DM-2853,"Put together a few slides for NCSA-IN2P3 meeting"," I put together a few slides for the NCSA-IN2P3 meeting describing previous scaling, middleware, and processing efforts of LSST DM. ",1 DM-2854,"Fix Qserv SsiSession worker race","The worker SsiSession implementation calls ReleaseRequestBuffer after handing the bound request to the foreman for processing. It therefore becomes possible for request processing to finish before ReleaseRequestBuffer is called by the submitting thread, resulting in a memory leak.",2 DM-2855,"Margaret's mgmt. activities in May","Weekly DMLT meeting Weekly ISO meeting Weekly NCSA local group meeting Met with NCSA networking person (Paul) to discuss progress and plans Attended End-to-End networking meeting Attended remotely the CCS-DAQ-OCS-DM Workshop for SCADA presentation by ISO Worked on/discussed AURA procurement contract amendment and to understand property management procedures at AURA and NCSA Discussed/planned agenda, made travel arrangements, prepared slides for trip to CC-IN2P3 2-day meeting in France with CC-IN2P3 group 4-day DMLT face-to-face meeting and T/CAM day Interview with candidate for systems lead Cleaned up Jira tickets from April Reviewed reporting requirements and Jira procedures with NCSA employee (Bruce) and managers (Doug, Brett) Attended several NCSA leadership development and training courses",22 DM-2856,"Multi-processing capability for shear test measurements","A suitable multi-cpu capability must be created for measurement tests. We are hoping to just use a pipe_task, but to do so, the butler must be customized to allow it to read our cutouts and psfs from galaxies and psfs generated from GalSim and PhoSim. This will be a relatively simple story if pipe_tasks running on 2 or 3 machines at UC Davis proves to be an adequate solution for running our shear experiments.",6 DM-2858,"Add support for ""ORDER BY f1, f2"" for has-chunks query","{code} QuerySession description: original: SELECT objectId, taiMidPoint FROM Source ORDER BY objectId, taiMidPoint ASC; has chunks: 1 needs merge: 1 1st parallel statement: SELECT objectId,taiMidPoint FROM LSST.Source_%CC% AS QST_1_ merge statement: SELECT objectId,taiMidPoint ORDER BY objectId,,taiMidPoint ASC ScanTable: LSST.Source {code} Merge statement has syntax error",4 DM-2859,"Return error for ""SELECT a FROM T ORDER BY b"" for has-chunks query","ORDER BY field has to be in result table => it has to be in select list. Return clear error message to user if not.",4 DM-2861,"Enquiry into MiniSub FO cable","Obtaining a quote from a company in Canada for a special clad cable for the Tololo-Pachon link",2 DM-2862,"RFI with vendors in Vina","Open day with all interested vendors to layout the projects for equipment on Mountain-Base and La Serena-Santiago links.",8 DM-2863,"Validate wmgr client / server versions ","If the client and server are on different versions, unexpected things can happen. (example: we run old version of the server, and use latest client). We need to check the version on both sides. ",4 DM-2864,"Fix bug related to selecting rows by objectId from non-director table","The following example illustrates the problem: Let's select one raw from qservTest_case01_qserv {code} select sourceId, objectId FROM Source LIMIT 1; +-------------------+-----------------+ | sourceId | objectId | +-------------------+-----------------+ | 29763859300222250 | 386942193651348 | +-------------------+-----------------+ {code} Then select it, but use ""sourceId"" in the query, all good here: {code} select sourceId, objectId FROM Source WHERE sourceId=29763859300222250; +-------------------+-----------------+ | sourceId | objectId | +-------------------+-----------------+ | 29763859300222250 | 386942193651348 | +-------------------+-----------------+ {code} But if we add ""objectId"", the row is not found: {code} select sourceId, objectId FROM Source WHERE sourceId=29763859300222250 and objectId=386942193651348; Empty set (0.09 sec) {code} Similarly, even without sourceId constraint, the query fails: {code} select sourceId, objectId FROM Source WHERE objectId=386942193651348; Empty set (0.09 sec) {code} ",8 DM-2865,"Merge BoundedField from HSC as is","To make headway on aperture corrections, we are bringing the HSC implementation of BoundedField over.",2 DM-2866,"Learn about Butler","Transferring knowledge from K-T to the DB team.",2 DM-2867,"Learn about Butler","Transferring knowledge from K-T to the DB team.",2 DM-2868,"Learn about Butler","Transferring knowledge from K-T to the DB team.",2 DM-2869,"Learn about Butler","Transferring knowledge from K-T to the DB team.",2 DM-2870,"Learn about Butler","Transferring knowledge from K-T to the DB team.",2 DM-2871,"Document butler v2 and transfer knowledge to Nate","Clean up and release prototype implementation of Butler v2.",9 DM-2872,"Setting with CoordKey doesn't support non-IcrsCoord arguments","Something in the {{FunctorKey}} template resolution doesn't allow {{Coord}} arguments to be used when setting record values with a {{CoordKey}} (only {{IcrsCoord}} arguments work.",1 DM-2873,"Handle ""where objectId between""","Query in a form: {code} select objectId from Object where objectId between 386942193651347 and 386942193651349 {code} currently fails with {code} ERROR 4120 (Proxy): Error during execution: -1 Ref=1 Resource(/chk/qservTest_case01_qserv/6630): 20150602-20:41:43, Complete (success), 0, Ref=2 Resource(/chk/qservTest_case01_qserv/6631): 20150602-20:41:43, State error (unrecognized), 0, Ref=3 Resource(/chk/qservTest_case01_qserv/6800): 20150602-20 (-1) {code} We already documented that such queries are not advised, but nethertheless qserv should handle it better, e.g., return a message ""not supported"" ",6 DM-2877,"demo package should contain the same comparison script used by CI","The lsst_dm_stack_demo package currently used in the CI system contains a {{bin/compare}} script that doesn't do all of the checks done by the numdiff script that buildbot runs. These need to be unified, so users can anticipate buildbot results and reproduce buildbot failures locally, especially when making changes to the expected-results file. While the {{numdiff}} script currently checks more columns than the {{compare}} script, I believe the compare script follows a much better approach and should be extended to be used in both places ({{numdiff}} converts everything to ascii then compares text files; {{compare}} works directly from the binary results and uses NumPy to do the comparisons).",2 DM-2879,"Add transformation tasks for new Butler dataset types","The new Butler dataset types created as part of the HSC deblender merge will need transformation tasks so they can be ingested to the database. See also DM-2191.",2 DM-2881,"Revisit Parser / IR","Revisit the existing parser code * consider reusing the code from maxscale or (antrl3) mysql parser from mysqlworkbench, or maybe reuse http://savage.net.au/SQL/sql-92.bnf.html and wrap in bison * separate IR node productions from grammar",94 DM-2882,"S15 Qserv CSS v2","Revisit Qserv Common State System. Implement mysql-based KV interface, and add support for updates. Implement ""locking"" mechanism. ",36 DM-2883,"wcslib is unable to read PTF headers with PV1_{1..16} cards","SCAMP writes distortion headers in form of PVi_nn (i=1..x, nn=5..16) cards, but this is rejected (correctly) by wcslib 4.14; there is a discussion at https://github.com/astropy/astropy/issues/299 The simplest ""solution"" is to strip the values PV1_nn (nn=5..16) in makeWcs() for CTYPEs of TAN or TAN-SIP and this certainly works. I propose that we adopt this solution for now. ",1 DM-2884,"LOG() macro fails if message is a simple std::string","lsst:log LOG() macro crash with fatal error if message is a simple string.",2 DM-2885,"Improve confusing error message","Selecting a column that does not exist results in confusing error. Example: {code} SELECT badColumnName FROM qservTest_case01_qserv.Object WHERE objectId=386942193651348; {code} ERROR 4120 (Proxy): Error during execution: -1 Ref=1 Resource(/chk/qservTest_case01_qserv/6630): 20150605-16:23:42, Error in result data., 1, (-1) Similarly, {code} select whatever FROM qservTest_case01_qserv.Object; {code} prints ERROR 4120 (Proxy): Error during execution: -1 Ref=1 Resource(/chk/qservTest_case01_qserv/6630): 20150605-16:23:52, Error in result data., 1, Ref=2 Resource(/chk/qservTest_case01_qserv/6631): 20150605-16:23:52, Error merging result, 1990, Cancellation requested Ref=3 Resource(/chk/qservTest_case01_qs (-1) (note, sourceId does not exist in Object table) ",5 DM-2887,"Fix broken IN - it now takes first element only","IN is broken - it only uses the first element from the list. Here is the proof: {code} select COUNT(*) AS N FROM qservTest_case01_qserv.Source WHERE objectId=386950783579546; +------+ | N | +------+ | 56 | +------+ 1 row in set (0.10 sec) mysql> select count(*) AS N FROM qservTest_case01_qserv.Source WHERE objectId=386942193651348; +------+ | N | +------+ | 39 | +------+ 1 row in set (0.09 sec) mysql> select COUNT(*) AS N FROM qservTest_case01_qserv.Source WHERE objectId IN(386942193651348, 386950783579546); +------+ | N | +------+ | 39 | +------+ 1 row in set (0.09 sec) mysql> select COUNT(*) AS N FROM qservTest_case01_qserv.Source WHERE objectId IN(386950783579546, 386942193651348); +------+ | N | +------+ | 56 | +------+ 1 row in set (0.11 sec) {code}",8 DM-2890,"isrTask assumes that the Exposure has a Detector","While trying to use the isrTask to interpolate over bad columns in PTF data I discovered that the code assumes that the Exposure has a Detector attached. Please remove this restriction. ",1 DM-2892,"Keep track of database of the director table","An L3 child table might very well have an LSST data release Object table as its director, while almost certainly not living in the DR database. To support it, we should keep track of the database name holding director's table. Note, this is related to DM-2864 - the code touched in that ticket should be checking the director's db name. Don't forget to add a unit test that will exercise it!",1 DM-2893,"Improve qproc unit testing framework","qproc unit testing framework allow to test the whole query analysis pipeline, it has grow and should be re-organized to be easilly understandable, maintainable.",5 DM-2895,"treat lsst_apps, lsst_libs and lsst_thirdparty as top level products not required by lsst_distrib","Per discussion on RFC-55, it was determined that lsst_apps and lsst_libs and lsst_thirdparty maybe be treated as separate top level products that lsst_distrib need not depend on them nor do they need to be included as part of CI builds.",1 DM-2897,"Travel to CCIN2P3 to establish realtionship","Travel to CC IN2P3. of establish operatal coordination between the sites",10 DM-2898,"Prototype file system loading tools for file system studies. ","worked on a tool chain to -- Extract file IO patterns from program strike. -- Represent in a flat file. Generate codes -- -- To make many dependent copies dummy files and directories -- To generate a python code to read and write files like the original application. -- To generate a c-code to read and write files like the original application. a driver program to run the pseudo codes in parallel. The system runs on toys and needs to be show to work on traces from ",10 DM-2899,"Host/Attend DM LT meeting ","host/attend DM LT meeting ",7 DM-2900,"Add queries that exercise non-box spatial constraints","Qserv has code to support: * qserv_areaspec_box * qserv_areaspec_circle * qserv_areaspec_ellipse * qserv_areaspec_poly but only the first one (box) is exercised in our integration tests. This story involves adding queries to test the other 3.",2 DM-2903,"Clean up gitolite","We need to clean up gitolite and cgit: * Repositories that have moved to GitHub should be removed (or, possibly, mirrored back from GitHub). * Empty repositories (like contrib/eups.git) and obsolete repositories (like LSST/DMS/afw_extensions_rgb.git) should be removed altogether. * contrib/data_products.git (the Data Products Definition Document source) should be moved to GitHub in the lsst org. * contrib/processFile.git should be moved to GitHub in the lsst-dm org unless the author adds some test cases and it can be integrated into the CI system as a top-level product (in which case it can go into the lsst org). * The primary authors of other contrib repositories should be contacted to see if they should be moved to GitHub in the lsst-dm or another org (possibly a new lsst-contrib org). In particular, contrib/plotz/* (Paul Lotz of the Telescope and Site subsystem) and contrib/pyreb (IN2P3 work for the Camera subsystem) contain current work that should be moved.",4 DM-2905,"Update Scons to v2.3.4","Scons has not been updated in over a year. RFC-61 agreed that we should upgrade it now before tackling some other {{scons}} issues.",1 DM-2908,"Add Gaussian PSF example to measurement task documentation","I see some documentation on how to add a placeholder Gaussian Psf to an image (to work around the fact that some algorithms require a Psf) was recently added to the release notes. I don't think that's actually appropriate, as the same algorithms also required Psfs in the framework - the failure mode was just different (previously, it'd just result in all objects being flagged and the peak position used for the centroid, so it may have been easy to miss - hence the change to a fatal error). I propose moving the example to the documentation for SingleFrameMeasurementTask, and taking it out of the release notes. I'll also make sure there's a link from the Measurement Framework Overhaul Release Notes page to the Doxygen for SingleFrameMeasurementTask - I'm not sure if that's sufficient to make up the visibility gap between the Doxygen docs and the release notes in Confluence, but I don't have any other short-term ideas.",1 DM-2909,"Remove unused code from sconsUtils","The code in {{deprecated.py}} in {{sconsUtils}} is not used by anything anywhere. [~jbosch] has indicated that the file can simply be removed.",1 DM-2910,"obs_cfht is broken with the current stack","obs_cfht's camera mapper is missing the new packageName class variable, so it is not compatible with the current stack. I suggest fixing obs_sdss and obs_subaru as well, if they need it.",1 DM-2911,"Build 2015_07 Qserv Release","See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe.",1 DM-2913,"Port HSC Curve-of-Growth code","Port content from HSC-1144, HSC-1236, HSC-1223, HSC-1219, HSC-1203, HSC-1153.",6 DM-2914,"Improve handling of extremely large blends","Port HSC code from issues: * [HSC-1250|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1250] * [HSC-1245|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1245] * [HSC-1237|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1237] * [HSC-1268|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1268] * [HSC-1274|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1274] * [HSC-1265|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1265] * [HSC-1228|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1228]",10 DM-2915,"Port safe coadd clipping from HSC","We have an algorithm on the HSC fork that modifies AssembleCoaddTask to clip outliers in a much safer way, based on detecting contiguous regions in the difference between a non-clipped coadd and an aggressively-clipped coadd, and only rejecting pixels that are outliers in a single epoch. One complication for this code transfer is that some of the coadd code has been refactored on the HSC side, and there may be code in hscPipe that duplicates much of what's in pipe_tasks. We may need help from [~price] to resolve those inconsistencies before tackling this issue. Once that's done, HSC code can be transferred from the following issues: - [HSC-1166|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1166] - [HSC-1202|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1202]",18 DM-2917,"obs_cfht unit tests are broken","obs_cfht has one unit test ""testButler"" that uses git://git.lsstcorp.org/contrib/price/testdata_cfht. 4 of the tests fail, as shown below. In addition, testdata_cfht is huge, and the tests barely use any of it. It's worth considering making a new test repo that is smaller, or if the amount of data is small enough, move it into afwdata or obs_cfht itself. {code} localhost$ tests/testButler.py CameraMapper: Loading registry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/DATA/registry.sqlite3 CameraMapper: Loading calibRegistry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/CALIB/calibRegistry.sqlite3 ECameraMapper: Loading registry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/DATA/registry.sqlite3 CameraMapper: Loading calibRegistry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/CALIB/calibRegistry.sqlite3 ECameraMapper: Loading registry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/DATA/registry.sqlite3 CameraMapper: Loading calibRegistry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/CALIB/calibRegistry.sqlite3 ECameraMapper: Loading registry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/DATA/registry.sqlite3 CameraMapper: Loading calibRegistry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/CALIB/calibRegistry.sqlite3 .CameraMapper: Loading registry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/DATA/registry.sqlite3 CameraMapper: Loading calibRegistry registry from /Users/rowen/LSST/code/testdata/testdata_cfht/CALIB/calibRegistry.sqlite3 E. ====================================================================== ERROR: testBias (__main__.GetRawTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/testButler.py"", line 122, in testBias self.getDetrend(""bias"") File ""tests/testButler.py"", line 110, in getDetrend flat = self.butler.get(detrend, self.dataId, ccd=ccd) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py"", line 218, in get location = self.mapper.map(datasetType, dataId) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/mapper.py"", line 116, in map return func(self.validate(dataId), write) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/cameraMapper.py"", line 287, in mapClosure return mapping.map(mapper, dataId, write) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py"", line 118, in map actualId = self.need(self.keyDict.iterkeys(), dataId) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py"", line 199, in need lookups = self.lookup(newProps, newId) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py"", line 345, in lookup return Mapping.lookup(self, properties, newId) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py"", line 168, in lookup where, self.range, values) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/registries.py"", line 120, in executeQuery c = self.conn.execute(cmd, values) OperationalError: no such column: extension ====================================================================== ERROR: testFlat (__main__.GetRawTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/testButler.py"", line 117, in testFlat self.getDetrend(""flat"") File ""tests/testButler.py"", line 110, in getDetrend flat = self.butler.get(detrend, self.dataId, ccd=ccd) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py"", line 218, in get location = self.mapper.map(datasetType, dataId) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/mapper.py"", line 116, in map return func(self.validate(dataId), write) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/cameraMapper.py"", line 287, in mapClosure return mapping.map(mapper, dataId, write) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py"", line 118, in map actualId = self.need(self.keyDict.iterkeys(), dataId) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py"", line 199, in need lookups = self.lookup(newProps, newId) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py"", line 345, in lookup return Mapping.lookup(self, properties, newId) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py"", line 168, in lookup where, self.range, values) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/registries.py"", line 120, in executeQuery c = self.conn.execute(cmd, values) OperationalError: no such column: extension ====================================================================== ERROR: testFringe (__main__.GetRawTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/testButler.py"", line 127, in testFringe self.getDetrend(""fringe"") File ""tests/testButler.py"", line 110, in getDetrend flat = self.butler.get(detrend, self.dataId, ccd=ccd) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py"", line 218, in get location = self.mapper.map(datasetType, dataId) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/mapper.py"", line 116, in map return func(self.validate(dataId), write) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/cameraMapper.py"", line 287, in mapClosure return mapping.map(mapper, dataId, write) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py"", line 118, in map actualId = self.need(self.keyDict.iterkeys(), dataId) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py"", line 199, in need lookups = self.lookup(newProps, newId) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py"", line 345, in lookup return Mapping.lookup(self, properties, newId) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/mapping.py"", line 168, in lookup where, self.range, values) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/10.1-3-g302a9ed/python/lsst/daf/butlerUtils/registries.py"", line 120, in executeQuery c = self.conn.execute(cmd, values) OperationalError: no such column: extension ====================================================================== ERROR: testRaw (__main__.GetRawTestCase) Test retrieval of raw image ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/testButler.py"", line 101, in testRaw raw = self.butler.get(""raw"", self.dataId, ccd=ccd, immediate=True) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py"", line 244, in get return callback() File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py"", line 242, in innerCallback(), dataId) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py"", line 238, in callback = lambda: self._read(pythonType, location) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/butler.py"", line 426, in _read location.getCppType(), storageList, additionalData) File ""/Users/rowen/LSST/lsstsw/stack/DarwinX86/daf_persistence/10.1-1-g6edbc00+1/python/lsst/daf/persistence/persistenceLib.py"", line 1430, in unsafeRetrieve return _persistenceLib.Persistence_unsafeRetrieve(self, *args) FitsError: File ""src/fits.cc"", line 1064, in lsst::afw::fits::Fits::Fits(const std::string &, const std::string &, int) cfitsio error: could not open the named file (104) : Opening file '/Users/rowen/LSST/code/testdata/testdata_cfht/DATA/raw/08BL05/w2.+2+2/2008-11-01/i2/1038843o.fits.fz[1]' with mode 'r' {0} lsst::afw::fits::FitsError: 'cfitsio error: could not open the named file (104) : Opening file '/Users/rowen/LSST/code/testdata/testdata_cfht/DATA/raw/08BL05/w2.+2+2/2008-11-01/i2/1038843o.fits.fz[1]' with mode 'r'' ---------------------------------------------------------------------- Ran 6 tests in 3.544s FAILED (errors=4) {code}",1 DM-2919,"PhotoCalTask mis-calling Colorterm methods","When I implemented DM-2797 I made a few errors in pipe_tasks: - PhotoCalTask mis-calls two methods of Colorterm by providing filterName, which is not needed - ColortermLibrary.getColorterm mis-handles glob expressions (the two arguments to fnmatch.fnmatch are swapped). We also need a unit test for applying colorterms, but that will require enough work that I have made a separate ticket for it: DM-2918. Meanwhile I have tested my changes by running Dominique's CFHT demo. This proves that the colorterm code runs, but does not prove that the terms are correctly applied.",1 DM-2920,"Clean up code in afw for Approximate background estimation","The intention is to eventually set {{useApprox=True}} (i.e. Chebychev Approximation) as the default for background estimation. However, in looking into the relevant code in afw/math while working on DM-2778, there is some clean-up and restructuring that needs to be done before resetting the defaults (which may also require adjusting some defaults in the calibrate stage to be more appropriate for the approximation, as opposed to inperpolation, scheme). This issue is to clean up the code and make sure it all operates coherently. A seperate ticket will be to actually reset the defaults and make any other config default changes required. In particular, the config setting of approxOrderX/binSize are not being assessed properly, nor is the behavior of the given undersampleStyle being executed. The under-sampling checks are currently only being done against the interpoation settings, which is not appropriate when useApprox=True. A temporary check was added in {{meas.algoritms.detection.getBackground()}} in DM-2778 so that it is currently ""safe"" to run with useApprox=True and any other user overridden config setting (binSize, approxOrder, undersampleStyle) (and there currently exists similar checks in {{pipe.tasks.matchBackground}}), but these should be removed once this issue has been implemented. ",3 DM-2921,"Set Approximation as default for background subtraction","Once the Approximate code in {{afw.math}} has been cleaned up (see DM-2920), set the default for background subtraction to be the Chebychev Approximation (i.e. useApprox=True). Ensure any other relevant config defaults (e.g. binSize, approxOrderX) are adjusted appropriately. This will change the outputs of the {{lsst_dm_stack_demo}}, so the ""expected"" files will need to be replaced along with this change in default settings (see DM-2778 for some comparisons of the demo outputs using the interpolation vs. approximation background estimation schemes).",1 DM-2922,"Initial DC Base dseign","Prepare a Document for the Initial Design of the Base and Summit Networks",6 DM-2923,"Port HSC-1199 to LSST (UNMASKEDNAN mask propagates to all amplifiers)","Port issue HSC-1199 to LSST stack to address UNMASKEDNAN mask propagates to all amplifiers",4 DM-2934,"Add RFD issue type to RFC project","To support the RFD process adopted in [RFC-53], an RFD issue type in the RFC project is required. While we could add RFD-specific fields to it, I think it's simplest if it's just generic with details provided in the Description.",1 DM-2927,"Modernize sconsUtils code to python 2.7 standard","As part of the work investigating DM-2839 I modernized the sconsUtils code to meet current coding standards (using {{in}} rather than {{has_key}}, using {{items()}} rather than {{iteritems}} etc). Since I'm highly doubtful that DM-2839 is going to be closed any time soon I will separate out the modernization patches into this ticket.",1 DM-2928,"move old ingest scripts into and retire old packages","This ticket implements RFC-57, by: - renaming datarel to daf_ingest (there is already a daf_ingest package, but it's *completely* empty, so I'll just force-push it all away) - removing everything from the renamed package that doesn't relate to ingest (including pruning dependencies) - removing ap and testing_endToEnd from the CI system ",1 DM-2929,"Some AFW tests are not enabled with no explanation","Running {{coverage.py}} on the AFW test suite indicated that two test classes in {{tests/wcs1.py}} are disabled. {{WCSTestCaseCFHT}} was added by [~rhl] in 2007 but disabled during a merge a long time ago by [~jbosch] in 2010 but with no indication as to why. {{WCSRotateFlip}} appeared in 2012 (added by [~krughoff]) but doesn't appear in the {{suite}} list at the end and so does not execute. Similarly {{testSchema.py}} has two tests that are not run: {{xtestSchema}} and {{testJoin}}. I assume {{xtestSchema}} is deliberately disabled but could there at least be a comment in the test explaining why? My feeling is that we should either run the tests or they should be removed. Having them their gives the impression they are doing something useful. Less importantly, {{warpExposure.py}} has some support code for comparing masked images that was written in 2009 by [~rowen] but which is not used anywhere in the test.",2 DM-2930,"Fix problem with Qserv related to restarting mysql","I noticed some strange (reproducible!) behavior: if I run: {code}qserv-check-integration.py --case=01{code} then restart mysqld {code}/etc/init.d/mysqld restart{code} then the query: {code}mysql --host=127.0.0.1 --port=4040 --user=qsmaster qservTest_case01_qserv -e ""SELECT COUNT(*) as OBJ_COUNT FROM Object WHERE qserv_areaspec_box(0.1, -6, 4, 6)""{code} consistently fails every single time. To fix it, it is enough to restart xrootd.",5 DM-2931,"We write truncated Wcs data to extended HDU tables in Exposures","When we write Wcs to extra HDUs in Exposures they are truncated if other than TAN/TAN-SIP. Please don't write them. A better long term solution is needed. In particular, we shouldn't be duplicating this information unnecessarily, and we need to be able to persist e.g. TPV to the tables so as to support CoaddPsf. These issues are not included here.",1 DM-2932,"Test install of OCS software on CentOS VMs","Install current version of the OCS software onto two VMs",24 DM-2935,"qserv-admin CREATE NODE fails","{noformat} qserv > CREATE NODE worker1 type=worker host=worker-1 port=5012 runDir=1; 06/15/2015 05:59:52 QADM ERROR: Missing parameter. (mysqlConn) ERROR: Missing parameter. (mysqlConn) {noformat} ",1 DM-2936,"Refactor Histogram in edu.caltech.ipac.visualize.plot package.","The Histogram has 6 constructors to handle 6 bitpixel data types which are byte, short integer, integer, long integer, float and double. Since FitsRead has now only works on float, there the Histogram should be refactored accordingly.",3 DM-2938,"CalibrateTask has an unwanted ""raise"" in it","On 2014-06-30 commit 696b641 a developer added a bare ""raise"" as a debugging aid to the CalibrateTask in pipe_tasks. That change was accidentally merged to master. I confirmed it was an accident and am filing this ticket as a way to remove the raise and run buildbot before merging to master.",1 DM-2939,"fix usage of obsolete astrometry interfaces in ProcessImageTask","As discussed recently on HipChat (Science Pipelines Standup), there's code in {{ProocessImageTask}} that assumes an ""astrometer"" attribute on a {{CalibrateTask}} instance. Since this is just needed to match a new set of sources against the reference catalog here, we should probably be using one of the new matcher objects, either by getting one from {{CalibrateTask}} via a documented interface, or by constructing a new one.",4 DM-2940,"DS9 tests fail if DS9 not running in some configurations","There are a few issues with the robustness of the {{testDs9.py}} tests in AFW. * The tests are skipped if the {{display_ds9}} package can not be loaded but they should also skip if {{ds9}} is missing or if {{ds9}} can not be loaded. The latter is especially important during builds that unset {{$DISPLAY}}. * The launching code in {{initDS9}} can not notice the simple case of {{ds9}} immediately failing to load. It simply assumes that there are delays in launch. The reason for this is that {{os.system}} does not return bad status if the command has been started in the background. Another scheme for starting {{ds9}} should be considered. Maybe a different exception could be raised specifically for failing to start it. * At the moment each test independently has a go at starting {{ds9}}. This makes the tests take a very long time (made worse by {{_mtv}} also trying multiple times) despite it being clear pretty quickly that {{ds9}} is never going to work. * Currently the {{mtv}} tests must run early as they are the only tests that attempt to start {{ds9}} if it is not running. If the two tests that call {{mtv}} are disabled two other tests fail. Ideally the {{initDS9}} code should be called in all cases.",1 DM-2945,"Wmgr refuses to serve queries from remote interface","Vaikunth discovered that wmgr returns 404 for all operations. It looks like wmgr can serve requests coming from 127.0.0.1 interface but returns 404 for queries from non-local interface.",1 DM-2948,"Remove explicit buildbot dependency on datarel","The buildbot scripts have an explicit dependency on the {{datarel}} package, which we'd like to remove from the stack. It uses {{datarel}} as the top-level product when building the cross-linked HTML documentation; {{lsstDoxygen}}'s {{makeDocs}} script takes a single package, and generates the list of packages to include in the Doxygen build by finding all dependencies of that package. So, to remove the explicit dependency on {{datarel}}, we need to either: - find a new top-level product with a Doxygen build to pass to {{makeDocs}} (e.g. by adding a trivial Doxygen build to {{lsst_distrib}}) - modify the argument parsing in {{lsstDoxygen}} to take a list of multiple products (it *looks* like the limitation to one package is only in the argument parsing), and pass it a list of top-level products in the buildbot scripts. This is currently a blocker for DM-2928, which itself a blocker for DM-1766, which has now been lingering for a few weeks now. I'm going to look for other ways to remove the block on the latter, but I don't have a solution yet.",3 DM-2949,"remove dead code and dependencies from datarel","Removing the {{datarel}} package entirely has proved to be difficult (DM-2928, DM-2948), so instead I'm simply going to remove non-ingest code (and dead ingest code) from the package, along with its dependencies on {{ap}} and {{testing_endToEnd}}. Other dependencies will be retained even if they aren't necessary for the code that will remain in {{datarel}}, to support {{lsstDoxygen}}'s use of {{datarel}} as a top-level package for documentation generation.",1 DM-2951,"Refactoring the class CropAndCenter","This class contains the codes which are not used. It needs to be simplified and refactored. ",4 DM-2952,"Crop needs to be refactored","This class needs to be refactored to be in consist with FitsRead class which treats all data type as float. Thus the bitpix in this class does not have to be treated based on its value.",3 DM-2953,"Qserv code cleanup and auto_ptr --> unique_ptr migration","Code cleanup, including migrating some parts to c++11 (in particular, auto_ptr --> unique_ptr)",4 DM-2954,"Add a unit test for aperture corrections in measurement task","DM-436 adds code to meas_base that allows one to run a subset of measurement algorithms based on execution order. This addition should have a unit test. DM-436 also tasks to measure and apply aperture correction. Those tasks should have unit tests.",6 DM-2955,"Setting up and running PhoSim for Psf Library","Debbie Bard leaving created some new work creating the Psf Libraries we need. While this od not a major task except for computer time, there is some setup required. I will get an account at SLAC and learn to run the PhoSim utilities she and Michael have developed. In the short term, Simon is going to do some runs for me. Meanwhile, I will get into Debbie's account and run her configuration at SLAC. The outputs then need to be checked to be sure that the Psfs are reasonable.",4 DM-2956,"Migrate Qserv code to nullptr",NULL,3 DM-2958,"Review ITIL V3.0 as prep for input to IT use case","ITIL is a standard breakdown of processes used in an IT system. While full ITIL may very well be too heavy LSST operations, it provides a useful checklist for he use cases being developed in the TOWG. I created an ITIL type spreadsheet to check against the dump of the workflows in the EA tool ",3 DM-2959,"add metric to application specific IO benchmarking tool","iosim is an application-specific benchmarking tool that is developed to be responsive to the request from LSST to select and investigate file systems ahead of actual benchmarks, and in advance of having workflow and other infrastructures needed to investigate file systems under realistic load. The week the software was -- Optimized to allow for faster test cycles. -- Threads were supported by squashing all thread IO into a single simulation process. -- Information from the original ""model"" program is propagated to the simulation program, allowing for comparison to the model. -- initial matplotlib plots allowing a degree of visualization of the performance of a simulated run was added The goals to deliver this capability in a few week when common systems at NCSA are available for testing.",4 DM-2960,"Management for Don for week of June 15","On boarded Mattais Carrasco-Kind to work in the process execution in the context of Level 3 processing. Misc.",4 DM-2961,"Port the psfextractor external library from HSC to LSST",NULL,14 DM-2962,"Prototype iRODS tiered resource with NERSC HPSS","iRODS can support access to a tape archive with the use of a ""tiered resource"" where one resource has the role of the cache, and a second has the role of archive. Use of such a tiered resource construct could be valuable to data management. Because a iRODS plugin for HPSS is readily available, we examine the set up and use of the tiered resource testing against NERSC HPSS.",4 DM-2964,"Read through and comment on latest version of LSE-78",NULL,1 DM-2966,"Design CSS that supports updates","Design how to redesign CSS, we currently take a snapshot when char starts. It is too static. ",2 DM-2967,"Fix to DM-2883 isn't quite right","The fix to DM-2883 (remove illegal PVi_j cards) isn't quite right, and the error was masked by a piece of code elsewhere that duplicated the functionality. The issues is that while PV1_[1-4] cards are indeed valid, the ones that SCAMP writes are not. So we should remove them too, if there are any other SCAMP TPV coefficients. The masking code was a unilateral removal of PVi_j cards dating back years. ",1 DM-2972,"Discourse evaluation (Part 1)","Work in support of evaluation Discourse as a DM platform for internal and external interactions.",3 DM-2975,"Quantify how much objects are blended","It would be useful to have a parameter that indicates how much any given galaxy is blended. This will be useful for testing how photometry or shears are affected by blending effects. Ports code from HSC-1260.",2 DM-2976,"SourceCatalog.getChildren requires preconditions but does not check them","This is a code transfer from HSC-1247.",2 DM-2977,"Miscellaneous CModel improvements from HSC","This improves handling of several edge case failure modes, tweaks the configuration to improve performance, and adds some introspection useful for Jose Garmilla's tests. Includes HSC-1288, HSC-1284, HSC-1228, HSC-1250, HSC-1264, HSC-1273, HSC-1240, HSC-1249, HSC-1238, HSC-990, HSC-1155, HSC-1191",2 DM-2978,"FootprintMerge: fix bug when identifying existing peaks in a merge.","If two separate footprints from the same catalog happen to be merged because an existing merged object overlaps both of them, the flags of which peaks are being detected in which bands is not being propagated. This is causing the apparent dropout of some sources in a merged catalog which were detected in single frame processing. Taken from ticket HSC-1270",1 DM-2980,"refactor coaddition code","The HSC fork has coaddition code in two places: pipe_tasks and hscPipe. The code in hscPipe is what we use (though that depends on the code in pipe_tasks in places), while the code in pipe_tasks is more similar to what's currently on the LSST side. We want to bring the refactored version in hscPipe back to LSST, but we want to put it directly in pipe_tasks to remove the code duplication that currently exists on the HSC side. Work on this issue should begin with an RFC that details the proposed changes. Note that this should not bring over the ""safe coadd clipping"" code, which is DM-2915.",5 DM-2981,"polygon masking in CoaddPsf","We need to create polygon-based masks of the usable area of the focal plane, persist them with exposure, and include them in coaddition of PSFs and aperture corrections. This includes HSC issues HSC-972, HSC-973, HSC-974, HSC-975, HSC-976. At least some of this will be blocked by DM-833, which is the port issue for coaddition of aperture corrections.",8 DM-2982,"Updating node status in qserv-admin to INACTIVE fails","In qserv-admin.py when attempting to update a node status from ACTIVE to INACTIVE the following error is produced: {code} > update node worker2 state=INACTIVE; Traceback (most recent call last): File ""/usr/local/home/vaikunth/src/qserv/bin/qserv-admin.py"", line 650, in main() File ""/usr/local/home/vaikunth/src/qserv/bin/qserv-admin.py"", line 645, in main parser.receiveCommands() File ""/usr/local/home/vaikunth/src/qserv/bin/qserv-admin.py"", line 163, in receiveCommands self.parse(cmd[:pos]) File ""/usr/l ocal/home/vaikunth/src/qserv/bin/qserv-admin.py"", line 180, in parse self._funcMap[t](tokens[1:]) File ""/usr/local/home/vaikunth/src/qserv/bin/qserv-admin.py"", line 380, in _parseUpdate self._parseUpdateNode(tokens[1:]) File ""/usr/local/home/vaikunth/src/qserv/bin/qserv-admin.py"", line 405, in _parseUpdateNode self._impl.setNodeState(**options) File ""/usr/local/home/vaikunth/src/qserv/lib/python/lsst/qserv/admin/qservAdmin.py"", line 660, in setNodeState self._kvI.set(nodeKey, state) File ""/usr/local/home/vaikunth/src/qserv/lib/python/lsst/qserv/css/kvInterface.py"", line 415, in set self._zk.set(k, v) File ""/usr/local/home/vaikunth/qserv/Linux64/kazoo/2.0b1+1/lib/python/kazoo-2.0b1-py2.7.egg/kazoo/client.py"", line 1170, in set return self.set_async(path, value, version).get() File ""/usr/local/home/vaikunth/qserv/Linux64/kazoo/2.0b1+1/lib/python/kazoo-2.0b1-py2.7.egg/kazoo/client.py"", line 1182, in set_async raise TypeError(""value must be a byte string"") {code}",1 DM-2983,"Backport HSC parallelization code","Assuming RFC-68 is approved, transfer the HSC code to LSST as described there.",4 DM-2984,"Compare LSST and HSC pipelines through ISR","Run both the HSC and LSST ISR routines on two to three visits worth of HSC engineering data. Compare the results. Where differences exist, either: * Create and work other tickets to resolve them; * Explain their origin and why we don't think they are a problem.",10 DM-2985,"Integrate javascript build with gradle","Integrate javascript build tools webpack with gradle.",2 DM-2986,"Conversion of FITS binary table extension to IPAC table format. ","FITS binary table contains data types and structures that cannot map directly to IPAC table. We need to define ways to handle these cases.",6 DM-2987,"Modify IpacTableParser to support extra wide table.","IpacTableParser fail to load IPAC table with extra wide headers and columns. Replace the logic for reading headers and columns information so that it will support any file/size.",2 DM-2989,"XY plot need to be able to handle multiple tables with the same name","XY plot was relying on a table request object to cache previously loaded tables. This was done for performance reason. However, table request is not reliable since the same request may be submitted multiple times.",2 DM-2990,"Add XYPlot to Python interface","Make it possible to add plots (not connected to a displayed table) to StandaloneUI. Add showXYPlot to python API.",6 DM-2991,"Firefly External Task Launcher","Implement and external task launcher, which forks a [python] process and gets back the results. The results can be a table, an image, or a JSON. ",10 DM-2992,"Search processors to get image, table, or json from an external task","Implement three search processors, which use the External Task Launcher (DM-2991): - to get a table (possibly in binary FITS format) - to get an image - to get JSON",8 DM-2993,"Products must not depend on anaconda","{{setupRequired(anaconda)}} should be removed from webservcommon.table. We want to keep the stack buildable with any python 2.7, and should not explicitly depend on anaconda.",1 DM-2996,"Understand and improve error code management","It seems there is several constants to store Qserv error code (for example see msgCode.h and util::ErrorCode, or MsgState::RESULT_ERR,JobStatus::RESULT_ERROR). This could certainly be simplified and clarified? Furthermore in util::Error it seems there's a confusion between code and statuses",8 DM-2997,"Bump eups anaconda package to 2.2","By popular request. ",1 DM-2998,"Begin to write a note for the TOWG using ITIL as a checklist","Began to work use cases, and found that they were long, considering the number r of aspects that need to be considered, Presented to the TOWG, got guidance to think in terms of processes, but to take that the effort estimates would have a reasonable basis. Got guidance to think about the sites's needs but to congress a central approach. Agree that this would be the struggle for the week.",4 DM-2999,"Management work for Don in the week of June 22","Internal and external recruiting. Input on NCSA re-organizaiton to ensure proper placement of LSST activities in the NCSA organization. Meetings.",3 DM-3001,"Whitepaper submission to NSF Cyber Summit","Working on drafting whitepaper and abstract for SCADA security challenges faced by LSST.",1 DM-3002,"ISO presentation to all-hands meeting","Presentation giving overview of ISO work, esp. w.r.t AUP and master security plan.",1 DM-3003,"Extend the Process Execution Framework to accomodate changes needed by SUI and others","Extend the Process Execution Framework to accomodate changes needed by SUI and others by changing the task and configuration classes. Covers effort from July 1st - August 31st at 0.25 FTE.",22 DM-3004,"Preliminary Process Execution Framework work","Preliminary work to extend the Process Execution Framework to accomodate changes needed by SUI and others. This story captures work done in June, prior to incorporating the activity into the baseline plan. Work will be logged under DM-3003 starting July 1st.",4 DM-3005,"prepare jenkins ""demo"" for usage as an interim CI system","We have a working plan of putting the buildbot-scripts under jenkins demo into usage as a production CI system as an intermediate step towards a fully decomposed build.",15 DM-3012,"complete puppet jenkins native type implimentation and merge upstream",NULL,20 DM-3016,"Meeting with CTSC at CLHS Portland OR","Discussed LSST security plan going forward. Specifically work on SCADA security plan. Meeting held at conference in Portland OR, June 14th, ACM CLHS.",1 DM-3019,"Do more research into Flux modules and bring one in ",NULL,16 DM-3020,"expose stretch to python API",NULL,2 DM-3021,"Improve region support","Some parts of the region support has been more testing because of the python interface. It is now clear what we should do.",4 DM-3022,"Convert Color Stretch dialog to React/flux/JavaScript",NULL,4 DM-3023,"Review LSE-78","Review either the current version #26 and/or the newest version when it become available.",2 DM-3024,"Discuss US WAN options with NCSA ESnet representative",NULL,1 DM-3025,"Provide network support for ceph and openstack lsst storage server efforts","work done to provide network connectivity, troubleshoot and monitor connections for the above efforts",1 DM-3026,"DLP/LDM-240 support chages"," - JIRA changes to create DLP project - lsst-sqre/sqre-jirakit to generate LDM-240-like display - iterate with T/CAMs, Kevin, Jeff",6 DM-3027,"Early access user onboarding and feedback ","Getting comments, testing, hipchat/JIRA changes",2 DM-3028,"Display stories in JIRA epic table display"," Solved with Issue Matrix plugin; unfortunately this removed the ""create issue in this epic"" functionality, so that needs to be a new ticket.",1 DM-3029,"Set up Slack for evaluation"," Free account procured and tested by various volunteers; next step is to apply for non-profit status which gives us the first paid tier free to 100 users. ",1 DM-3030,"Set up Discourse for evaluation."," Server up on DO at community.lsst.org. Email needs fixing before volunteer users can be invited. ",1 DM-3031,"Addressing File corruption in iRODS 4.1.x","We examine solutions for repairing corrupt files within an iRODS 4.1.x zone.",2 DM-3032,"Read revised LSE-209 and LSE-70","Read over the revised LSE-209 and LSE-70 documents",4 DM-3033,"Add Sdss3Mapper to ingest, convert and map SDSS-III ""frame"" files","SDSS-III does not use the fpC file format for science images. Science images are now released as [""frame"" files. | http://data.sdss3.org/datamodel/files/BOSS_PHOTOOBJ/frames/RERUN/RUN/CAMCOL/frame.html] The primary science image (hdu0) comes background subtracted and calibrated to units of nanomaggies, with the backgrounds and flat-field conversions included as extensions. The astrometric information is in hdu3 instead of a separate asTrans file. obs_sdss should be able to ingest frame files and map them to load as dataset ""raw."" It should also optionally replace the backgrounds and de-calibrate to convert the units back from nanomaggies to counts. This will be implemented as lsst.obs.sdss.sdssMapper.Sdss3Mapper.",5 DM-3034,"Margaret's mgmt. activities in June",NULL,14 DM-3035,"Check czar->proxy messages size","These messages are stored in VARCHAR(255) (FYI, MEMORY tables can't contain TEXT). We just need to make sure we have a reasonable fixed size CHAR (and maybe check whether we are hitting the limit, and log it somewhere)",4 DM-3036,"Move Qserv code comment to LSST documentation standards","LSST documentation standards: https://confluence.lsstcorp.org/display/LDMDG/Documentation+Standards#DocumentationStandards-RequiredDocumentationStyle is different from the previous standards used by Qserv (i.e. /// text). We should convert everything to LSST documentation standards.",6 DM-3037,"remove lsst/log wrapper from Qserv","lsst/log API looks stable now, so removing the wrapper would simplify the code.",1 DM-3038,"Debug problems with Qserv at scale",NULL,20 DM-3090,"Implement test suite for new class SqlTransaction","Some test that shows that transactions are properly committed/aborted would be nice to have.",1 DM-3091,"Remove unused function populateState() ","Qserv doesn't seem to relaunch no more chunk query in case it fails (see DM-2643) And this function is now unused: {code:bash} qserv@clrinfopc04:~/src/qserv (master)$ grep -r populateState core/ core/modules/qdisp/Executive.cc:void populateState(lsst::qserv::qdisp::ExecStatus& es, {code} ",1 DM-3093,"LOE - Week ending 7/10/15",NULL,1 DM-3094,"LOE - Week ending 7/17/15",NULL,1 DM-3095,"LOE - Week ending 7/24/15",NULL,2 DM-3096,"LOE - Week ending 7/31/15",NULL,2 DM-3097,"Bi-weekly meeting with Victor and Iain.",NULL,2 DM-3098,"Incident response report template",NULL,1 DM-3099,"Incident response security work plan document",NULL,2 DM-3101,"Creation of XML descriptions of messages sent to OCS","Create XML descriptions of messages sent to the OCS. Upload these to a new github repository.",4 DM-3102,"Resolve segmentation fault in LoggingEvent destructor","There seems to be a possible race condition in log4cxx::spi::LoggingEvent::~LoggingEvent. I've had multiple segmentation faults in that function. In all cases, another thread was involved in writing. In at least 2 cases, the second thread was in XrdCl::LogOutFile::Write. ",5 DM-3103,"RFI with prospective DWDM vendors for Chile National Networks","Hold Request for Information meetings in Chile with equipment vendors. ",12 DM-3104,"Add ""ORDER BY"" clause to lua SQL query on result table","If user query has ""ORDER BY"", then lua can't just execute ""SELECT * FROM result"" because the order for such query is not guaranteed. To fix that, we need to add ""ORDER BY"" clause to the ""SELECT * FROM result"" query on the lua side. Once we have the above, we might want to remove ""ORDER BY"" from the query class which runs a merge step on the czar (this has to be done in query analysis step).",8 DM-3105,"Add assertXNearlyEqual methods for image-like classes","Presently one can compare two image-like objects using free functions imagesDiffer, masksDiffer and maskedImagesDiffer in lsst.afw.image.testUtils. These should be replaced by assertXNearlyEqual methods that afw adds to lsst.utils.tests, as per DM-2193. If necessary, we could leave the old functions around for awhile. But I would prefer to simply get rid of them if we can. One subtlety is that the current functions take numpy arrays, not afw image-like class instances. Examine the existing users of the code to determine how best to deal with that.",4 DM-3106,"Add slot for calibration flux","This is a port of [HSC-1005|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1005].",2 DM-3108,"Use aperture flux for photometric calibration","This is a port of work performed on HSC but without a ticket. Relevant commits are: * [05bef6|https://github.com/HyperSuprime-Cam/meas_astrom/commit/05bef629adc37e44ea8482aab88e2eb38a47e3a0] * [4a6be5|https://github.com/HyperSuprime-Cam/meas_astrom/commit/4a6be51c53f61e70f151de7f29863cb723197a99] * [69d35a|https://github.com/HyperSuprime-Cam/obs_subaru/commit/69d35a890234e37c1142ddbeff43e62fe36e6c45] * [9c996d|https://github.com/HyperSuprime-Cam/obs_subaru/commit/9c996d75c423ce03fb54c4300d9c7561b5c1ea99]",1 DM-3109,"Add support for accessing schema from QueryContext","When we are analyzing a query, sometimes there are situations where we need to know the schema of tables involved in a query. It will also be useful for checking if user is authorized to run query, and for queries like ""SHOW CREATE TABLE"". This story involves writing code that will provide access to schema.",3 DM-3110,"qserv code cleanup","I made some random cleanup of the qserv code while playing with css v2. I want to push these changes to master, thus I am creating this story for this. It involves improvements to logging in UserQueryFactory and Facade (both are now per-module), removing unnecessary namespace qualifiers, and whitspace cleanup.",1 DM-3111,"Convert major portion of GWT in Firefly to pure JavaScript (W16)","Continue to convert GWT portion of Firefly to pure JavaScript",100 DM-3114,"Enable aperture correction in the integration test","The present integration test does not enable aperture correction. This should be enabled and the results sanity-checked. This is a separate ticket rather than DM-436 at Jim Bosch's suggestion, to avoid ticket bloat. It requires two separate changes: - update obs_sdss's SdssCalibrateTask to measure and apply aperture correction - update the expected results from the integration test lsst_dm_stack_demo",4 DM-3118,"Attend CCS-DAQ-OCS-DM Workshop IV",NULL,6 DM-3119,"whitepaper CFP for nsf cyber summit",NULL,2 DM-3120,"Execution Framework prototype",NULL,8 DM-3121,"Graphical communication interface","Creating a graphical representation of execution framework",2 DM-3125,"basic monitoring of jenkins nodes with notification","This last weekend, the build slaves el6-2 and el7-2 ran out of disk space and were causing stack-os-matrix build failures. We should have an active monitoring system that sends notifications via at least one of hipchat/email/pagerduty. There is disk utilitization information present in jenkins itself, aws cloudwatch, and ganglia (as of v0.2.x of the demo). However, it may make be more convenient (read: expedient) to use a dedicated monitoring system such as sensu instead of mining existing data sources. We should also investigate if we can configure jenkins to not schedule jobs on a slaves with low disk space.",6 DM-3126,"gcc 4.8 package does not create a symlink bin/cc","I created a new lsst package named ""gcc"" that contains Mario's gcc 4.8 package. I used it to build lsst_distrib on lsst-dev and it worked just fine. Unfortunately the package does not include bin/cc (which should be a symlink to bin/gcc), and this is wanted because the LSST build system uses cc to build C code. The desired fix is to modify the installer to make a symlink bin/cc that points to bin/gcc.",2 DM-3133,"add ""dax_"" prefix to data access related packages","As agreed at [Data Access Mtg 2015/07/13|https://confluence.lsstcorp.org/display/DM/Data+Access+Meeting+2015-07-13], add dax_ prefix towebserv, webservcommon, webserv_client, dbserv, imgserv, metaserv",1 DM-3134,"Implement Result Streaming in Qserv","Qserv supporting streaming results while the query is running to client application.",60 DM-3135,"FY19 Handling unexpected conditions during query execution","Detect and handle unexpected conditions during query execution (e.g. bad chunk, hit per user resource limit)",53 DM-3136,"Add & use new mask plane for out-of-bounds regions","Add a new mask plane for regions with no data - fully vignetted, edge patches in coadd. This is a port of [HSC-669|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-669].",2 DM-3137,"Handle bad pixels in image stacker","We currently OR together all mask bits, but we need to be cleverer about how we handle pixels that are bad in some but not all inputs. This is a port of work carried out on [HSC-152|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-152].",1 DM-3138,"Open day for RFP from Equipment vendors","Open day with vendors to describe the needs and requirements of the mountain to base and La Serena to Santiago networks. This is the first formal meeting in the procurement process. Vendors will be invited to propose their solution along with cost.",8 DM-3139,"HSC backport: extra ""refColumn"" class attributes in multiband","This is a transfer for changesets for [HSC-1283|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1283]. ",1 DM-3140,"add gcc to list of packages in lsstsw","Add gcc to the list of packages in etc/repos.yaml in lsstsw",1 DM-3141,"Reduce verbosity of astrometry","The astrometry.net solver that runs by default in meas_astrom 10.1 is very verbose. Here's an example running HSC data with an SDSS reference catalog: {code} $ processCcd.py /tigress/HSC/HSC --output /tigress/pprice/lsst --id visit=904020 ccd=49 --clobber-config : Loading config overrride file '/home/pprice/LSST/obs/subaru/config/processCcd.py' WARNING: Unable to use psfex: No module named extensions.psfex.psfexPsfDeterminer hscAstrom is not setup; using LSST's meas_astrom instead Cannot import lsst.meas.multifit: disabling CModel measurements Cannot import lsst.meas.extensions.photometryKron: disabling Kron measurements Cannot enable shapeHSM ('MEAS_EXTENSIONS_SHAPEHSM_DIR'): disabling HSM shape measurements Cannot import lsst.meas.extensions.photometryKron: disabling Kron measurements Cannot enable shapeHSM ('MEAS_EXTENSIONS_SHAPEHSM_DIR'): disabling HSM shape measurements : Loading config overrride file '/home/pprice/LSST/obs/subaru/config/hsc/processCcd.py' : input=/tigress/HSC/HSC : calib=None : output=/tigress/pprice/lsst CameraMapper: Loading registry registry from /tigress/pprice/lsst/_parent/registry.sqlite3 CameraMapper: Loading calibRegistry registry from /tigress/HSC/HSC/CALIB/calibRegistry.sqlite3 processCcd: Processing {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904020, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 49, 'expTime': 30.0} processCcd.isr: Performing ISR on sensor {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904020, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 49, 'expTime': 30.0} processCcd.isr WARNING: Cannot write thumbnail image; hsc.fitsthumb could not be imported. afw.image.MaskedImage WARNING: Expected extension type not found: IMAGE processCcd.isr: Applying linearity corrections to Ccd 49 processCcd.isr.crosstalk: Applying crosstalk correction afw.image.MaskedImage WARNING: Expected extension type not found: IMAGE : Empty WCS extension, using FITS header processCcd.isr: Set 0 BAD pixels to 647.04 processCcd.isr WARNING: There were 6192 unmasked NaNs processCcd.isr WARNING: Cannot write thumbnail image; hsc.fitsthumb could not be imported. processCcd.isr: Flattened sky level: 647.130493 +/- 12.733898 processCcd.isr: Measuring sky levels in 8x16 grids: 648.106765 processCcd.isr: Sky flatness in 8x16 grids - pp: 0.024087 rms: 0.006057 processCcd.calibrate: installInitialPsf fwhm=5.88235294312 pixels; size=15 pixels processCcd.calibrate.repair: Identified 80 cosmic rays. processCcd.calibrate.detection: Detected 303 positive sources to 5 sigma. processCcd.calibrate.detection: Resubtracting the background after object detection processCcd.calibrate.initialMeasurement: Measuring 303 sources (303 parents, 0 children) processCcd.calibrate.astrometry: Applying distortion correction processCcd.calibrate.astrometry: Solving astrometry LoadReferenceObjects: read index files processCcd.calibrate.astrometry.solver: Number of selected sources for astrometry : 258 processCcd.calibrate.astrometry.solver: Got astrometric solution from Astrometry.net LoadReferenceObjects: getting reference objects using center (1023.5, 2084.5) pix = Fk5Coord(320.3431396, 0.5002365, 2000.00) sky and radius 0.00194896 rad LoadReferenceObjects: search for objects at Fk5Coord(320.3431396, 0.5002365, 2000.00) with radius 0.111667372351 deg LoadReferenceObjects: found 495 objects LoadReferenceObjects: trimmed 257 out-of-bbox objects, leaving 238 processCcd.calibrate.astrometry.solver: Fit WCS: use iter 2 because it had less linear scatter than the next iter: 0.307471 vs. 0.320229 pixels processCcd.calibrate.astrometry: 186 astrometric matches processCcd.calibrate.astrometry: Refitting WCS processCcd.calibrate.astrometry: Astrometric scatter: 0.047945 arcsec (with non-linear terms, 174 matches, 12 rejected) processCcd.calibrate.measurePsf: Measuring PSF /tigress/HSC/LSST/stack10_1/Linux64/anaconda/2.1.0-4-g35ca374/lib/python2.7/site-packages/numpy/core/_methods.py:59: RuntimeWarning: Mean of empty slice. warnings.warn(""Mean of empty slice."", RuntimeWarning) /tigress/HSC/LSST/stack10_1/Linux64/anaconda/2.1.0-4-g35ca374/lib/python2.7/site-packages/numpy/core/_methods.py:71: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) /home/pprice/LSST/meas/algorithms/python/lsst/meas/algorithms/objectSizeStarSelector.py:143: RuntimeWarning: invalid value encountered in less update = dist < minDist processCcd.calibrate.measurePsf: PSF star selector found 163 candidates processCcd.calibrate.measurePsf: PSF determination using 114/163 stars. processCcd.calibrate.repair: Identified 92 cosmic rays. processCcd.calibrate: Fit and subtracted background processCcd.calibrate.measurement: Measuring 303 sources (303 parents, 0 children) processCcd.calibrate.astrometry: Applying distortion correction processCcd.calibrate.astrometry: Solving astrometry processCcd.calibrate.astrometry.solver: Number of selected sources for astrometry : 258 Solver: Arcsec per pix range: 0.153025, 0.18516 Image size: 2054 x 4186 Quad size range: 205.4, 4662.78 Objs: 0, 50 Parity: 0, normal Use_radec? yes, (320.343, 0.500178), radius 1 deg Verify_pix: 1 Code tol: 0.01 Dist from quad bonus: yes Distractor ratio: 0.25 Log tune-up threshold: inf Log bail threshold: -230.259 Log stoplooking threshold: inf Maxquads 0 Maxmatches 0 Set CRPIX? no Tweak? no Indexes: 3 /tigress/HSC/astrometry_net_data/sdss-dr9-fink-v5b/sdss-dr9-fink-v5b_and_263_0.fits /tigress/HSC/astrometry_net_data/sdss-dr9-fink-v5b/sdss-dr9-fink-v5b_and_263_1.fits /tigress/HSC/astrometry_net_data/sdss-dr9-fink-v5b/sdss-dr9-fink-v5b_and_263_2.fits Field: 258 stars Quad scale range: [641.674, 2208.56] pixels object 1 of 50: 0 quads tried, 0 matched. object 2 of 50: 0 quads tried, 0 matched. object 3 of 50: 0 quads tried, 0 matched. object 4 of 50: 0 quads tried, 0 matched. object 5 of 50: 0 quads tried, 0 matched. object 6 of 50: 0 quads tried, 0 matched. Got a new best match: logodds 787.099. log-odds ratio 787.099 (inf), 178 match, 1 conflict, 75 distractors, 220 index. RA,Dec = (320.343,0.500213), pixel scale 0.167612 arcsec/pix. Hit/miss: Hit/miss: ++-+++++-++++++++++++--++-+--+++++-+-+++++++++-+++++++-+++++-+++++++++++++-++++++-++++++-+++++++-+++ Pixel scale: 0.167612 arcsec/pix. Parity: pos. processCcd.calibrate.astrometry.solver: Got astrometric solution from Astrometry.net LoadReferenceObjects: getting reference objects using center (1023.5, 2084.5) pix = Fk5Coord(320.3431396, 0.5002365, 2000.00) sky and radius 0.00194896 rad LoadReferenceObjects: search for objects at Fk5Coord(320.3431396, 0.5002365, 2000.00) with radius 0.111667328272 deg LoadReferenceObjects: found 495 objects LoadReferenceObjects: trimmed 257 out-of-bbox objects, leaving 238 processCcd.calibrate.astrometry.solver: Fit WCS: use iter 2 because it had less linear scatter than the next iter: 0.306732 vs. 0.320115 pixels processCcd.calibrate.astrometry: 186 astrometric matches processCcd.calibrate.astrometry: Refitting WCS processCcd.calibrate.astrometry: Astrometric scatter: 0.048271 arcsec (with non-linear terms, 174 matches, 12 rejected) processCcd.calibrate.photocal: Not applying color terms because config.applyColorTerms is False processCcd.calibrate.photocal: Magnitude zero point: 30.685281 +/- 0.058711 from 173 stars processCcd.calibrate: Photometric zero-point: 30.685281 processCcd.detection: Detected 1194 positive sources to 5 sigma. processCcd.detection: Resubtracting the background after object detection processCcd.deblend: Deblending 1194 sources processCcd.deblend: Deblended: of 1194 sources, 143 were deblended, creating 358 children, total 1552 sources processCcd.measurement: Measuring 1552 sources (1194 parents, 358 children) processCcd WARNING: Persisting background models processCcd: Matching icSource and Source catalogs to propagate flags. processCcd: Matching src to reference catalogue LoadReferenceObjects: getting reference objects using center (1023.5, 2087.5) pix = Fk5Coord(320.3429016, 0.5001781, 2000.00) sky and radius 0.00195667 rad LoadReferenceObjects: search for objects at Fk5Coord(320.3429016, 0.5001781, 2000.00) with radius 0.112109149864 deg LoadReferenceObjects: found 499 objects LoadReferenceObjects: trimmed 261 out-of-bbox objects, leaving 238 processCcd.calibrate.astrometry.solver: Fit WCS: use iter 1 because it had less linear scatter than the next iter: 0.300624 vs. 0.300652 pixels {code} The verbosity of the astrometry module is out of proportion with the rest of the modules, which makes it difficult to follow the processing. This is a pull request for fixes I have made.",1 DM-3142,"Port HSC optimisations for reading astrometry.net catalog","Some astrometry.net catalogs used in production can be quite large, and currently all of the catalog must be read in order to determine bounds for each component. This can make the loading of the catalog quite slow (e.g., 144 sec out of 177 sec to process an HSC image, using an SDSS DR9 catalog). We have HSC code that caches the required information, making the catalog load much faster. The code is from the following HSC issues: * [HSC-1087: Make astrometry faster|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1087] * [HSC-1143: Floating point exception in astrometry|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1143] * [HSC-1178: Faster construction of Astrometry.net catalog|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1178] * [HSC-1179: Assertion failure in astrometry.net|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1178] While there have been some changes to the LSST astrometry code that will mean we can't directly cherry-pick the HSC code, yet I think the main structure remains, so the approach can be copied without much effort.",3 DM-3144,"Audit and improve warm-start configuration options","Many of our command-line tasks - particularly the high-level MPI-based drivers we're moving over from the HSC side - typically reuse intermediate data products they find on disk rather than regenerate them by default, and have a suite of configuration options to control this behavior. While this aids in faster reprocessing of aborted or failed jobs, it can produce results users would consider surprising (""I re-ran with a new version of the pipeline and nothing changed""), and (IMO) should not be the default behavior for any task. We also need to guarantee that any warm-start reprocessing always produces the exact same results as a single consistent run. I do not believe this is currently the case for some of the options in ProcessCcdTask. Finally, we should put these configuration options somewhere other than the main task config, because they control how the processing is done, not what the results are, and hence should not be checked against existing config files in an output data repo before running. This will probably require a new mechanism in pipe_base; perhaps a second config class associated with each Task, containing only options that affect the ""how"" of processing without affecting the ""what"". The first step of implementing this issue should be an RFC.",8 DM-3146,"Purchase transceivers for use by AURA","Cisco Xenpack for extra long distance 10Gbs",5 DM-3147,"Install switches and transceivers on Pachon and Tololo","Buy switches to deploy MPLS over the fibers. ",8 DM-3148,"Configure switches for AURA tenants","Switch configuration for use by individual tenants of AURA",8 DM-3151,"CI validation of lsstsw's repos.yaml","Having some sort of automatic ""lint check"" of the repos.yaml file is desirable due to the length of time required to do a full up test of lsstsw. It should be possible to cobble a sanity checker together that can be run from travis-ci.",1 DM-3153,"meas_base still uses eups in tests","{{tests/centroid.py}} uses EUPS to determine the location of the data file used by the test. This needs to be fixed to use a location relative to the test file.",1 DM-3154,"meas_astrom still using eups in tests","In DM-2636 we modified the tests to be skipped if EUPS is not available. I've had a closer look and all the ones I have glanced at seem to be easily fixable to run without EUPS. The tests seem to be using EUPS to locate the {{meas_astrom}} (effectively asking EUPS for the location of the test file), then a path to the astrometry.net test data within the {{tests/}} directory is located and then EUPS is asked to setup {{astrometry_net_data}} using that path. Since the table files are all empty this is the equivalent to simply assigning the {{ASTROMETRY_NET_DATA_DIR}} environment variable directly to the path in the tests sub-directory. Making this change to one of the tests seems to work so I will change the rest.",2 DM-3155,"W16 Qserv Release and Testing","This epic captures stories related to building, testing and maintaining Qserv releases, along with related documentation. Note that testing involve running larger scale tests more or less monthly to ensure we haven't broken anything. 3 SPs per month.",23 DM-3158,"Data Distribution + Qserv",NULL,75 DM-3160,"Improve name and default value of MeasureApCorrConfig.refFluxAlg","The config name refFluxAlg should be refFluxField (since it is a flux field name prefix) and the default should be base_CircularApertureFlux_5 instead of base_CircularApertureFlux_0 (thus giving a reasonable radius instead of one that is ridiculously too small). I should have handled it on DM-436 but it slipped through.",1 DM-3161,"Implement MySQL-based KVInterface","This story covers adding mysql-based implementation of KVInterface. The implementation will be done in C++, and it will be exposed to the python layer.",12 DM-3162,"Extend KVInterface - add support for updates","The CSS Facade and KVInterface currently do not support updates. This story covers adding support for basic updates.",4 DM-3163,"CSS/QMeta interaction in czar","CSS currently does not have any notion of locks. The snapshots of CSS should be taken per query, for each query and they should be done in coordination with Query Metadata. This will ensure tables used by a running query never gets altered or deleted while the query is running.",10 DM-3164,"S17 Fine-tune Data Access Interfaces",NULL,15 DM-3167,"Install the LSST Stack on loaned laptop ",NULL,5 DM-3170,"Qserv Release and Testing","This epic captures stories related to building, testing and maintaining Qserv releases, along with related documentation. Note that testing involve running larger scale tests more or less monthly to ensure we haven't broken anything. 3 SPs per month.",9 DM-3171,"X16 Qserv Refactoring",NULL,29 DM-3172,"Accessing the current obs_decam package ","Installing the non-official obs_decam package from Simon Krughoff's Github and processing some DECam data blindly ",14 DM-3173,"In CalibrateTask if one disables psf determination then aperture correction will fail","In pipe_tasks CalibrateTask, by default aperture correction uses source flag ""calib_psfUsed"" to decide if a source is acceptable to use for measuring aperture correction. If PSF determination is disabled then this flag is never set and aperture correction will fail with a complaint that there are 0 sources. ",1 DM-3174,"CalibrateTask instantiates measureApCorr, applyApCorr and photocal subtasks using the wrong schema","CalibrateTask instantiates measureApCorr, applyApCorr and photocal subtasks using the initial schema ""schema1"" instead of the final schema. Normally this would not matter since most of the fields are shared, but aperture correction wants aperture flux at a larger radius than the narrowest option, and schema1 may only provide the narrowest option. In any case it is safer to instantiate those three subtasks using the final schema, since they are only ever run on the final schema. (Several other subtasks are run on both the initial and final schema, and should continue to be instantiated using schema1).",1 DM-3175,"Build 2015_08 Qserv Release","See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe.",1 DM-3176,"Build and Test 2015_09 Qserv Release","See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe.",3 DM-3177,"Build and Test 2015_10 Qserv Release","See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe.",3 DM-3178,"Build and Test 2015_11 Qserv Release","See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe.",3 DM-3179,"Build and Test 2015_12 Qserv Release","See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe.",3 DM-3180,"Build and Test 2016_01 Qserv Release","See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe.",3 DM-3181,"Build and Test 2016_02 Qserv Release","See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe.",1 DM-3182,"Aperture correction not applied for some measurements","Aperture correction needs to be applied every time a measurement is run after it is first measured in CalibrateTask. As of DM-436 aperture correction is only being applied in CalibrateTask, which for example means the information is overwritten during the final measurement of ProcessImageTask.run. This is probably best done by adding code to apply aperture correction to BaseMeasurementTask, so it is inherited by SingleFrameMeasurementTask and ForcedMeasurementTask.",5 DM-3183,"CalibrateTask instantiates some subtasks with the wrong schema","CalibrateTask instantiates some subtasks with the wrong schema, in particular: - astrometry is instantiated with the final schema but run on schema1 - measureApCorr, applyApCorr and photocal are instantiated with schema1 but run on the final schema One way this can cause problems is that schema1 may not have the data needed to measure aperture correction (e.g. it may contain only one tiny radius of aperture flux), as came to light when running the lsst stack demo.",1 DM-3185,"Investigate workflow for OpenStack via Python scripts","We investigate the use of Python scripts that work against OpenStack APIs to start up VMs and configure them for use, for example, in processing, build & test scenarios, etc. We are initially working against the ISL OpenStack, and intend to test against the ""nebula"" system when it becomes available. (This type if work was initiated in issues DM-1787, DM-1788 in a previous Epic, and we continue within the context of DM-1273, ) ",12 DM-3186,"Add PT.12 Filter/Science_Ccd_Exposure tables to extend test query coverage","Filter table is missing from case02, case05 data, so next query can't be tested: {code:sql} -- datasets/case02/queries/3023_joinObjectSourceFilter.sql.FIXME --- Join on Source and Filter and select specific filter in region --- https://dev.lsstcorp.org/trac/wiki/db/Qserv/IN2P3/BenchmarkMarch2013 --- https://dev.lsstcorp.org/trac/wiki/db/queries/007 - SELECT objectId, taiMidPoint, fluxToAbMag(psfMag) FROM Source JOIN Object USING(objectId) JOIN Filter USING(filterId) -WHERE ra_PS BETWEEN 1 AND 2 -- noQserv AND decl_PS BETWEEN 3 AND 4 -- noQserv --- withQserv qserv_areaspec_box(1,3,2,4) AND filterName = 'u' AND variability BETWEEN 0 AND 2 {code} Same thing for case02:1011_objectsForExposure and case02:1030_timeSeries.sql",4 DM-3188,"Fix UDF for case01 query: 3005_orderByRA.sql","query {code:bash} mysql --host=127.0.0.1 --port=4040 --user=qsmaster --batch qservTest_case01_qserv -e ""SELECT * FROM Object WHERE qserv_areaspec_box(0.,1.,0.,1.)"" {code} returns nothing whereas {code} SELECT * FROM Object WHERE ra_PS BETWEEN 0. AND 1. -- noQserv AND decl_PS BETWEEN 0. AND 1. {code} does (but doesn't use geom index)",6 DM-3189,"Remove _chunkId, _subChunkId column from case02:Object table","This columns are Qserv internal and shouldn't be in input data. For example, this prevents case02:3021_selectObjectSortedByRA to work. Check also that these columns aren't in other test data set and remove FIXME suffix from related broken query.",4 DM-3190,"Document deprecation of DecoratedImage","According to discussion on Hipchat (20 July 2015) {quote} Jim Bosch: [...] DecoratedImage is strongly deprecated, though {quote} This was news to me, and certainly isn't reflected [in (at least the obvious place) in Doxygen|https://lsst-web.ncsa.illinois.edu/doxygen/x_masterDoxyDoc/afw_sec_image.html]. It should be.",1 DM-3192,"Re-implement watcher based on new CSS implementation","Current watcher implementation (in {{admin/bin/watcher.py}}) is based on direct watching of zookeeper updates via kazoo. If we are to re-implement CSS based on mysql then watcher needs to be updated to support it. Mysql does not have watch mechanism, so it has to be done via polling or using some other mechanism if synchronous notifications are needed.",8 DM-3193,"Audit existing test and development system","Document in the wiki how the test and development systems are connected and configured.",3 DM-3194,"Fix cluster install procedure and improve docker support","Document how-to update cluster from Qserv release: See http://www.slac.stanford.edu/exp/lsst/qserv/2015_07/HOW-TO/cluster-deployment.html",1 DM-3196,"makeWcs() chokes on decam images in 10.1","In 10.0, processCcdDecam.py could process decam images to completion (whether the WCS was read correctly is a different question). Now it fails on makeWcs() (see traceback below), and I suspect this change in behavior is related to DM-2883 and DM-2967. Repository with both data and code to reproduce: http://www.astro.washington.edu/users/yusra/reproduce/reproduceMakeWcsErr.tar.gz (apologies for the size) The attachment is a document describing the WCS representation in the images from the community pipeline, courtesy of Francisco Forster. Please advise. This ticket captures any changes made to afw. {code} D-108-179-166-118:decam yusra$ processCcdDecam.py newTestRepo/ --id visit=0232847 ccdnum=10 --config calibrate.doPhotoCal=False calibrate.doAstrometry=False calibrate.measurePsf.starSelector.name=""secondMoment"" doWriteCalibrateMatches=False --clobber-config : Loading config overrride file '/Users/yusra/lsst_devel/LSST/repos/obs_decam_ya/config/processCcdDecam.py' : Config override file does not exist: '/Users/yusra/lsst_devel/LSST/repos/obs_decam_ya/config/decam/processCcdDecam.py' : input=/Users/yusra/decam/newTestRepo : calib=None : output=None CameraMapper: Loading registry registry from /Users/yusra/decam/newTestRepo/registry.sqlite3 processCcdDecam: Processing {'visit': 232847, 'ccdnum': 10} makeWcs WARNING: Stripping PVi_j keys from projection RA---TPV/DEC--TPV processCcdDecam FATAL: Failed on dataId={'visit': 232847, 'ccdnum': 10}: File ""src/image/Wcs.cc"", line 130, in void lsst::afw::image::Wcs::_initWcs() Failed to setup wcs structure with wcsset. Status 5: Invalid parameter value {0} lsst::pex::exceptions::RuntimeError: 'Failed to setup wcs structure with wcsset. Status 5: Invalid parameter value' Traceback (most recent call last): File ""/Users/yusra/lsst_devel/LSST/DMS5/DarwinX86/pipe_base/10.1-3-g18c2ba7+49/python/lsst/pipe/base/cmdLineTask.py"", line 320, in __call__ result = task.run(dataRef, **kwargs) File ""/Users/yusra/lsst_devel/LSST/DMS5/DarwinX86/pipe_base/10.1-3-g18c2ba7+49/python/lsst/pipe/base/timer.py"", line 118, in wrapper res = func(self, *args, **keyArgs) File ""/Users/yusra/lsst_devel/LSST/repos/obs_decam_ya/python/lsst/obs/decam/processCcdDecam.py"", line 77, in run mi = exp.getMaskedImage() File ""/Users/yusra/lsst_devel/LSST/DMS5/DarwinX86/daf_persistence/10.1-1-g6edbc00+28/python/lsst/daf/persistence/readProxy.py"", line 41, in __getattribute__ subject = oga(self, '__subject__') File ""/Users/yusra/lsst_devel/LSST/DMS5/DarwinX86/daf_persistence/10.1-1-g6edbc00+28/python/lsst/daf/persistence/readProxy.py"", line 136, in __subject__ set_cache(self, get_callback(self)()) File ""/Users/yusra/lsst_devel/LSST/DMS5/DarwinX86/daf_persistence/10.1-1-g6edbc00+28/python/lsst/daf/persistence/butler.py"", line 242, in innerCallback(), dataId) File ""/Users/yusra/lsst_devel/LSST/DMS5/DarwinX86/daf_persistence/10.1-1-g6edbc00+28/python/lsst/daf/persistence/butler.py"", line 236, in location, dataId) File ""/Users/yusra/lsst_devel/LSST/repos/obs_decam_ya/python/lsst/obs/decam/decamMapper.py"", line 118, in bypass_instcal wcs = afwImage.makeWcs(md) File ""/Users/yusra/lsst_devel/LSST/DMS5/DarwinX86/afw/10.1-26-g9124caf+1/python/lsst/afw/image/imageLib.py"", line 8706, in makeWcs return _imageLib.makeWcs(*args) RuntimeError: File ""src/image/Wcs.cc"", line 130, in void lsst::afw::image::Wcs::_initWcs() Failed to setup wcs structure with wcsset. Status 5: Invalid parameter value {0} lsst::pex::exceptions::RuntimeError: 'Failed to setup wcs structure with wcsset. Status 5: Invalid parameter value' {code} ",2 DM-3199,"Standardize Qserv install procedure: step 1 build docker container for master/worker instance and development version ","- shmux could be used for parallel ssh (remove Qserv builtin one) - look at ""serf and consul"" (See Confluence pages) - improve doc: http://www.slac.stanford.edu/exp/lsst/qserv/2015_07/HOW-TO/index.html - run multiple instances/versions of Qserv using different run dir/ports and the same data",8 DM-3202,"Include reference magnitude errors in PhotoCal","PhotoCal task currently ignores the uncertainties on reference sources, which can lead to problems when the reference and measured catalogs have relatively little overlap or otherwise disagree on how trustworthy a source is.",3 DM-3204,"W16 Data Access and Db Release Documentation","Write Release documentation covering Data Access and Database work.",5 DM-3205,"Revisit cost of replicating non-partitioned tables on all nodes","Revisit size of all non-partitioned tables, and cost of replicating them on all worker nodes.",4 DM-3206,"Estimate I/O load for non-partitioned tables","Estimate realistic IO load from user queries on non-partitioned tables. Consider whether there might be hot spots (eg., maybe a small subset of columns from exposure is used very often. If it is it, maybe it'd be worth replicating only these columns across all worker nodes and serve the rest from one shared file system).",6 DM-3207,"Experiment with CONNECT engine for non-partitioned tables","Idea: store non-partitioned tables in a dedicated mysql server, and bring them to the worker nodes using connect engine. This story involves exploring if that would work, and uncovering potential pitfalls.",10 DM-3209,"Add debugging for astrometry.net solver","To be able to debug astrometric matching, it helps to be able to visualise the source positions, the distorted source positions, and the reference positions. This is a pull request to add these.",1 DM-3212,"Query Coverage",NULL,28 DM-3213,"Get ImageDifferenceTask running again","ImageDifferenceTask doesn't run. The issues I've seen so far are related to the measurement overhaul. This ticket will capture the one-off updates needed to get this running again. Appropriate Bugs and Papercuts epic?",10 DM-3214,"ChebyshevBoundedField should use _ not . as field separators for persistence","ChebyshevBoundedField uses ""."" instead of ""\_"" as field separators in its afw table persistence. This is the old way of doing things, and unfortunately causes errors when reading in older versions of tables, becaus afw converts ""."" to ""_"" in that situation. This shows up as a unit test failure in DM-2981 (brought over from HSC) when an older version table is read in. It is an open question whether to fix this as part of DM-2981 (which conveniently has a test that shows the problem, though not intentionally so) or separately, in which case a new test is wanted. In the former case I'm happy to do the work so I can finish DM-2981. Many thanks to Jim Bosch for diagnosing the problem.",1 DM-3215,"Begin drafting specification document for the Level 1 System","Don Petravick (.5 FTE) + Jason Alt (.8 FTE) + Paul Wefel (.125 FTE) July 2015 - August 2015",50 DM-3216,"Initial work to process DECam data with LSST stack","Hsin-Fang Chiang (1 FTE) July 2015 - August 2015",80 DM-3217,"Extrapolate to the current document ","Discussed use cases in the context of the TOWG, and also with the site and telescope group. A picture of the structure of IT operations has (I believe consensus emerged) that is the four layers ITIL cake Service Design (cataloged, budget, availability, etc). Service Transition (The work of inserting Charge into the system). Service Delivery (the work of running the as-is system of servinces) ITC -- The work of providing the Facility, Hardware and networking. I also obtained the ability to interact with EA. (but still working to master it and its concepts)",8 DM-3218,"unable to create public images","Errors are returned when attempting to upload an image marked as public.",1 DM-3223,"Improve czar-worker communication debugging","Add features to make it easier to debug communication problems. Particularly, record the source of a message, and remove extraneous messages.",2 DM-3224,"Document setting up multi-node Qserv and running integration test",NULL,4 DM-3227,"openstack API endpoint is broken","Similar to what was observed in DM-3226, the referral endspoint returned by {code:java} https://nebulous.ncsa.illinois.edu:5000 {code} are not FQDNs. This fundamentally breaks any attempt to use the API one step past authenticating with keystone. This is an example HTTP response: {code:java} HTTP/1.1 200 OK Date: Mon, 27 Jul 2015 23:11:02 GMT Server: Apache/2.4.10 (Ubuntu) Vary: X-Auth-Token X-Distribution: Ubuntu x-openstack-request-id: req-ac7bb613-86ef-43ab-a663-75c2ed3fb124 Content-Length: 1656 Content-Type: application/json {""access"": {""token"": {""issued_at"": ""2015-07-27T23:11:02.342216"", ""expires"": ""2015-07-28T00:11:02Z"", ""id"": ""99b843d4baf94569a0d34ca4fecb470c"", ""tenant"": {""description"": null, ""enabled"": true, ""id"": ""d1f16653856540d386224fb057b5b00c"", ""name"": ""LSST""}, ""audit_ids"": [""fAP8851vTQi1n5pYmNoIjw""]}, ""serviceCatalog"": [{""endpoints"": [{""adminURL"": ""http://nebula:9292"", ""region"": ""RegionOne"", ""internalURL"": ""http://nebula:9292"", ""id"": ""49365a8e8fe743af9d517e84a98e3ee9"", ""publicURL"": ""http://nebula:9292""}], ""endpoints_links"": [], ""type"": ""image"", ""name"": ""glance""}, {""endpoints"": [{""adminURL"": ""http://nebula:8774/v2/d1f16653856540d386224fb057b5b00c"", ""region"": ""RegionOne"", ""internalURL"": ""http://nebula:8774/v2/d1f16653856540d386224fb057b5b00c"", ""id"": ""c1e31df3656042ef9c5502efd7d574f2"", ""publicURL"": ""http://nebula:8774/v2/d1f16653856540d386224fb057b5b00c""}], ""endpoints_links"": [], ""type"": ""compute"", ""name"": ""nova""}, {""endpoints"": [{""adminURL"": ""http://nebula:9696"", ""region"": ""RegionOne"", ""internalURL"": ""http://nebula:9696"", ""id"": ""266c9dc8e0344f8fa3f078652e868443"", ""publicURL"": ""http://nebula:9696""}], ""endpoints_links"": [], ""type"": ""network"", ""name"": ""neutron""}, {""endpoints"": [{""adminURL"": ""http://nebula:35357/v2.0"", ""region"": ""RegionOne"", ""internalURL"": ""http://nebula:5000/v2.0"", ""id"": ""5d474008bcee4f44800546e3f3302404"", ""publicURL"": ""http://nebula:5000/v2.0""}], ""endpoints_links"": [], ""type"": ""identity"", ""name"": ""keystone""}], ""user"": {""username"": ""jhoblitt"", ""roles_links"": [], ""id"": ""6ea0c8e153b04ae29572c5fd877b6ac3"", ""roles"": [{""name"": ""user""}], ""name"": ""jhoblitt""}, ""metadata"": {""is_admin"": 0, ""roles"": [""142761bd922e453294e9b7086a227cbc""]}}} {code} ",1 DM-3228,"evaluate NCSA OpenStack against SQRE requirements and provide feedback - part 1","See also https://confluence.lsstcorp.org/pages/viewpage.action?spaceKey=LDMDG&title=NCSA+Nebula+OpenStack+Issues",2 DM-3230,"Refinement, restatement of DM facilites and functions ","Began detailed refinement. The initial version was a word document with omnigraffle figures. Began moving this to LSST Confluence to make the model more accessible for scrutiny. The top level functional - physical breakdown diagrams now exist and transferring the functional breakdown from word to confluence is in place. I have request that the simple citation package be installed in the LSST lira (not perfect, but helps). ",10 DM-3232,"Potential talk for All-Hands",NULL,1 DM-3233,"Potential talk for nsf cyber summit",NULL,1 DM-3234,"Collab. with Ron Lambert and Oliver W.",NULL,2 DM-3237,"Fix problems with no-result queries on multi-node setup","For queries like: select * from Object where id = qserv can't map it to any chunk, and it ends up executing SELECT * FROM qservTest_case01_qserv.Object_1234567890 AS QST_1_ WHERE objectId= the chunk 1234567890 is a special chunk and it exists on all nodes. And that fails with: (build/qdisp/QueryResource.cc:61) - Error provisioning, msg=Unable to write file; multiple files exist. code=2 ",1 DM-3239,"Build a demo system for camera team to use the Firefly external task launcher","We have the code to launch ab external task from Firefly server. ( see DM-2991) IN order to facilitate the development by the UIUC group for Camera team, we need to have a simple example to show how to connect the front extension to the external task at the server side. ",10 DM-3240,"Support to the camera team development","We need someone to attend the weekly meeting at UIUC group for camera team. To discover issues and answer questions. This is an on-going effort. ",10 DM-3241,"Create images for the mask bits at server side","LSST FITS images will have a extension that indicate the mask bits. In order to overlay the masks on the primary image, we need to turn the mask bits into a set of images. This task is to take the requested bits and FITS as input, output a set of images for each requested bit. Each bit will have different color. ",20 DM-3242,"a simple demo version to use Firefly in iPython notebook","We want to build a simple demo version of Firefly that works in the iPython notebook. It will make it easier for users to try out Firefly visualization capabilities with Python APIs. ",10 DM-3243,"Include polygon bounds in CoaddPsf logic","This is a port of [HSC-974|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-974]. Original description: The {{CoaddPsf}} class should use the polygon bounding areas that were added to {{Exposure}} and {{ExposureRecord}} in DM-2981 (was: HSC-973) when determining which PSF images to coadd.",1 DM-3245,"Add support for passing query classification info from user to czar","We need to be able to pass information from user about query type (sync/async). This will require tweaking the parser. ",8 DM-3246,"Add support for async query results","Modify Qserv to support async queries: send query results to the right place instead of to mysql proxy. In this story, we can simply use some reasonable default location and send the results there. Later on we will extend qserv to make it configurable.",10 DM-3247,"Add support for configuring async queries","Extend Qserv configuration to allow a DBA to specify (a) where results from async queries should be stored and (b) what rules to apply when purging old results. Note that we need to think about the purging rules, it is not immediately obvious what would make most sense.",5 DM-3249,"Revisit and document user-facing aspects of async queries","Outline all aspects of async queries that are affecting users, discuss with the DM team, and document. This includes things like: * managing async queries (checking status, terminating) * retrieving results from async queries * managing query results (purging policies etc) * probably more, need to think about it...",8 DM-3253,"Unify KVInterface python and c++ interfaces","Swig the C++ mysql-based KvInterface implementation. ",8 DM-3254,"FY18 Qserv Health Verification Tool","Need a tool for verifying if all the services are up and running, including things like whether udfs are loaded",79 DM-3255,"Prepare and implement RFP for DWDM devices","Prepare and implement RFP",10 DM-3256,"Fiber path Gate to pachon","Discussion on fiber path",5 DM-3257,"Port flux.scaled from HSC","[HSC-1295|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1295] introduces {{flux.scaled}}, which measures the flux within a circular aperture that is set from the size of the PSF, scaled by some factor. Stephen Gwyn recommends using this as our fiducial calibration flux.",2 DM-3258,"CoaddPsf.getAveragePosition() is not a valid position","This is a port of [HSC-1138|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1138] to LSST. That is an aggregate of two related minor fixes: * {{CoaddInputRecorder}} should default to {{saveVisitGoodPix=True}} so that average positions in the {{CoaddPsf}} can be properly weighted; * {{computeAveragePosition}} and {{doComputeKernelImage}} should be consistent about the data included when determining whether a source is off image.",1 DM-3259,"Define polygon bounds for CCDs based on vignetted regions","This is a port of [HSC-976|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-976] to LSST. The original issue description was: We should set the polygon bounds (added in DM-2981 [was HSC-973]) for HSC CCD exposures to cover the non-vignetted regions. This should probably be done in ISR or some other camera-specific location. Note that, contrary to the description in DM-2981, this functionality was not included there.",1 DM-3261,"Fix problems in xrootd discovered in multi-node qserv tests",NULL,15 DM-3262,"Explore how to run multi-node tests","Not testing qserv code often enough in multi-node environment led to introducing many problems over the past two years since we last run large scale test. It should be simple for developer to run a multi-node test. This story covers work related to understanding how to run integration test on multi-node.",4 DM-3263,"W16 Make Query Cancellation Robust","It is clear that the code responsible for query cancellation needs some more thoughts and refactoring (it was prematurely rushed when Daniel was about to leave slac). In particular, the code seems to have some subtle problems. We need to debug these problems and solve them.",49 DM-3264,"Audit & cherry-pick HSC-1126 fixes","[HSC-1126|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1126] contains a number of unrelated bug fixes. Given the nature of that ticket, it's not immediately clear which might already have been ported to LSST, which don't apply, and, of the others, what dependencies they have on code which might still be in the queue for merger. We need to dig through that ticket and ensure that everything is properly merged.",2 DM-3265,"add task to meas_astrom to fit an aribtrary WCS with a TAN-SIP WCS","Sometime in the past, Russell Owen wrote a method to take an arbitrary WCS and approximate it as a TanWcs (our implementation of FITS' TAN-SIP WCS formalism). That method is currently just a utility function in the unit test testFitTanSipWcsHighOrder.py. This issue will promote that method to a full-fledge task in meas_astrom.",2 DM-3266,"Reconfigure Openstack systems","1) Configured the IPMI on 13 systems.",22 DM-3345,"Administrative - 7-2015",NULL,2 DM-3346,"Networking support of Openstack efforts",NULL,2 DM-3347,"assertWcsNearlyEqualOverBBox and friends is too hard to use as a free function","assertWcsNearlyEqualOverBBox and similar functions elsewhere in afw were written to be methods of lsst.utils.tests.TestCase, so their first argument is a testCase. This is fine for use in unit tests, but a hassle to use as free functions because the user must provide a testCase argument (though it need only be a trivial class with a fail(self, msgStr) method). Worse, that minimal requirement is not documented, so technically providing a simple mock test case is unsafe. I have two proposals: - Document the fact that testCase need only support fail(self, msgStr). This makes it clear how to safely use these functions as free functions. - Allow testCase to be None, in which case RuntimeError is raised. That makes these functions even easier to use as free functions. ",1 DM-3348,"Include translation of aliases in measurement calibration","Tasks that calibrate measurement outputs should include transferring (and translating, as needed) the aliases in the original measurement catalog. This should include transferring slots, which may involve ""renaming"" the original slots, as those names refer explicitly to raw measurements (i.e. ""slot_PsfFlux"" might become ""slot_PsfMag"").",3 DM-3349,"Add test case for ExposureRecord::contains","In DM-3243 we ported from HSC the ability to take account of the associated {{validPolygon}} when checking whether a point falls within an {{Exposure}}. This functionality was not accompanied by an adequate unit test.",2 DM-3351,"Reproducing errors of the current obs_decam package","Learning the stack and development worlflow by reproducing errors in obs_decam as DM-3196. Changes from the stack need to be incorporated into obs_decam package to keep the package up to date, hence the errors. As a learning process I reproduced the errors and used her fix (afw branch u/yusra/DM-3196) to move on. ",2 DM-3352,"Study basic afw ","Learn the basic operation of the aft package about handling images, tables, etc. ",6 DM-3353,"Management until end july 215","Finsh system development lead and on Onboard Jason Alt. Deal with management of group providing NCSA openstack. Review and re-review procurement contract prose and directions. Coordinate TOWG input with NCSA management. Gain acume about exiting design documents, decide and prototype details refinement/ analysis /restatement needed to manage project at NCSA. Management of people.",10 DM-3354,"Read in the FITS cube that Herschel project produced","IRSA needs to be able to read in the FITS cube generated by Herschel project. We need to support and guide the effort so the code is generic enough for non-Herschel data. ",2 DM-3355,"Support the FITS cube reader","RSA needs to be able to read in the FITS cube generated by Herschel project. We need to guide the effort so the code is generic enough for non-Herschel data.",1 DM-3356,"Fix Firefly build script so it'll work with latest version of gradle","Firefly build was failing when using gradle version 2.5. Minor changes to the dependencies declaration fixed it.",1 DM-3357,"Margaret's mgmt. activities in July",NULL,30 DM-3358,"Add mysql-based test to multi-node integration test","At the moment multi-node integration test runs only on multi-node using Qserv, it does not run on plain mysql, and thus we can't validate results. The story involves tweaking qserv_testdata such that we can run mysql test on the czar, and compare results from mysql and qserv.",5 DM-3359,"Investigate jenkins creating a container per job",NULL,2 DM-3362,"Run large scale tests",NULL,8 DM-3363,"Debug problem with joins in multi-node tests","We seen to have problems with joins: {quote} SELECT o.deepSourceId, s.objectId, s.id, o.ra, o.decl FROM Object o, Source s WHERE o.deepSourceId=s.objectId; {quote} cluster seems to have hung. I can send new queries to the czar, and they show up in the czar's log, but they don't get answered (they can be cancelled). Cancelling the join works(at least for the czar) but no further queries work.",4 DM-3364,"Analyze qserv performance / KPIs",NULL,5 DM-3365,"Explore Qserv authentication and authorization",NULL,8 DM-3366,"Produce Data Access & DB team S15 Release docs","Complete these documents: * https://confluence.lsstcorp.org/display/DM/Summer+2015+Qserv+Release * https://confluence.lsstcorp.org/display/DM/Summer+2015+WebServ+Release",4 DM-3367,"Add multi-process python runner script for Galaxy Shear Experiments","The current runner scripts are in tcsh and bash. There is no good excuse for this, except that it was easy to implement. Since we need both multi-threading and better parameter parsing, this will be replaced with a python script.",2 DM-3368,"Port HSC MPI driver for single-visit processing","Transfer the {{reduceFrames.py}} script and the {{ProcessExposureTask}} it utilizes from hscPipe to a new package in the LSST stack (RFC-68 proposes calling this new package {{pool_tasks}}, but this isn't set in stone). We should probably rename either the driver script or the task (or both), so they agree; the lack of consistency is a historical artifact on the HSC side, and I think it's time to change that.",6 DM-3369,"Port HSC MPI driver for coaddition","Port the HSC driver for coaddition, {{stack.py}} from hscPipe to a new LSST package (the same as DM-3368). In the process, we should remove the inclusion of {{ProcessCoaddTask}}, and instead run detection and background subtraction only. I think it might be time to consider renaming this task as well; I find it a little unfortunate we use ""coadd"" everywhere else but ""stack"" here.",4 DM-3370,"Port HSC MPI driver for multi-band coadd processing","Port the HSC MPI driver of multi-band coadd processing, multiBand.py, from hscPipe to a new LSST package (the same as in DM-3368).",4 DM-3371,"Port HSC --rerun option for CmdLineTask","Port the HSC side's {{--rerun}} option for specifying processing inputs and outputs. This work should be preceded by an RFC; we've proposed implementing this option on the LSST side in the past, and it was met with some resistance as it isn't strictly necessary. We've since found it extremely convenience on the HSC side, and I think it's very much worth porting.",4 DM-3372,"Port, replace, or defer HSC-side provenance of EUPS products","The HSC pipeline checks that setup EUPS products are identical between runs with the same output directory, in the same way configuration is checked in both the LSST and HSC pipelines. The implementation is a bit messy, and it's not strictly necessary, so it's not clear we should port this over as-is, or just wait for a better implementation to be provided by the Process Middleware team. We should at least RFC this question now.",6 DM-3373,"Port HSC code for generation of calibration products","Port HSC code for building calibration products (flats, bias frames, etc.).",6 DM-3374,"add realistic Footprints to measurement code","The current measurement code for the galaxy shear simulations uses the full postage stamp bounding box for the Footprint. We need to use more realistic Footprints for some of the tests we want to run. That probably involves running {{SourceDetectionTask}} and somehow combining that with the input-catalog based iterating already in the measurement code.",4 DM-3375,"Test shear bias vs. CModel region.nGrowFootprint","One piece of how the CModel code chooses its fit region size is via nGrowFootprint, which is used to grow the original detection Footprint. We should test how changing this parameter affects the _m_ and _c_ shear biases between input and recovered shear. They should decrease for larger nGrowFootprint values, and eventually plateau. We want to find the point where this happens, and see how the parameter affects both the fit region area and the shear biases before this threshold. It may be necessary to make a small modification to the CModel code to output the fit region area to complete this test.",4 DM-3376,"Test shear bias vs. CModel region.nInitialRadii","Like DM-3375, but testing the region.nInitialRadii parameter instead. This parameter sets the fit region using a multiple of the half-light ellipse from an initial approximate fit. The full fit region is formed as the union of this with the grown detection Footprint, so it may be necessary to set nGrowFootprint to a negative number to see any affect from this parameter.",4 DM-3377,"Initial issue investigation for the nebula openstack"," The nebula openstack system at NSCA first became available ~Fri Jul 24 and the week of Jul 27 -- 31 was spent testing and debugging issues that the LSST team identified within, for example, DM-3225, DM-3219, DM-3227 and others. ",20 DM-3378,"NSF Cyber Summit talk","NSF Cyber Security Summit talk: a case study of LSST cyber security. Talk goes over challenges and successes with LSST's security program. Talk is divided into four sections: security plan, data security, user access, and security for the observation site.",7 DM-3380,"Port HSC hooks for simulated source injection","Port HSC hooks injecting simulated sources into real images to test processing. This includes the code in {{fakes.py}} in pipe_tasks and its callers. The pipeline does not include code for actually adding the fake sources; it just provides a callback interface that is implemented by third-party plugins such as https://github.com/clackner2007/fake-sources.",4 DM-3381,"Add test cases for thresholding","In DM-3136 changes were made to the way thresholds are handled in detection ([{{a4b011d}}|https://github.com/lsst/meas_algorithms/commit/a4b011dd0775908c925ad9f40f802f9ed8723ef9] and [{{74c2ed0}}|https://github.com/lsst/meas_algorithms/commit/74c2ed0b79afce4c94b0db5f1e168c28ba1aa15b]). These were not accompanied by test cases, but they should be.",2 DM-3382,"security playbook","Practical document for handling and responding to incidents.",1 DM-3383,"Meeting with on HTCondor","Attended meeting with Miron Livny.",1 DM-3384,"Port HSC improvements to HSM moments code","The HSM shear estimation has received several improvements and important bugfixes on the HSC side that need to be ported to LSST. This is complicated by the fact that much of the code has been entirely rewritten on the LSST side to work within the new measurement framework, but we've also synchronized this package with the HSC side much more frequently than with other packages.",4 DM-3385,"Administrative - 8-2015",NULL,1 DM-3386,"Parallelism Framework migration",NULL,2 DM-3387,"Make use of good pixel count when building CoaddPsfs","When building a CoaddPsf we have the ability to take account of the number of pixels contributed by the inputs (see http://ls.st/paj and DM-3258). However, the {{CoaddPsf}} constructor fails to use this information. It should copy this field when copying the provided {{ExposureCatalog}}, so that {{computeAveragePosition}} can use it.",1 DM-3388,"Edit end-to-end test plan to reflect current DM plans",NULL,2 DM-3390,"Re-generate data for large scale tests at in2p3","Sources were incorrectly duplicated, need to be redone",3 DM-3391,"Refactor Zscale.java class ","In early this year, the decision all data types would be converted to float in FitsRead. Thus,the bitpixel is not relevant. In Zscale, it still uses bitpixel to test the data type. It should be refactored in the same manner as FitsRead etc. ",2 DM-3392,"Fix precision related problem in UDFs","SciSQL udfs seem to have a subtle precision problem. The following query that is not relying on scisql returns one row: {quote} select ra, decl, deepSourceId FROM Object o WHERE decl between 0.992 and 0.993 and ra between 19.171 and 19.172; +------------------+-------------------+------------------+ | ra | decl | deepSourceId | +------------------+-------------------+------------------+ | 19.1719166801441 | 0.992087616433663 | 4368217963236477 | +------------------+-------------------+------------------+ 1 row in set (2 min 9.49 sec) {quote} But equivalent scisql-based query: {quote} select ra, decl, deepSourceId FROM Object o WHERE qserv_areaspec_box(0.992, 19.171, 0.993, 19.172); {quote} will fail to find that row. If we relax the search criteria just a little bit, it finds some other row, but still not the one with decl = 0.992087616433663 {quote} select ra, decl, deepSourceId FROM Object o WHERE qserv_areaspec_box(0.99, 19.171, 0.999, 19.172); +-------------------+-----------------+------------------+ | ra | decl | deepSourceId | +-------------------+-----------------+------------------+ | 0.994098536926311 | 19.171425377618 | 4372684729224984 | +-------------------+-----------------+------------------+ {quote} ",4 DM-3393,"Fix column names in query result","The following shows the problem (See the column names in the results, they are not what user will expect). It happens for all aggregates: min, max, avg, count etc {code} select min(ra_PS), min(decl_PS), max(ra_PS), max(decl_PS), avg(ra_PS) from Object; +----------------+---------------+---------------+---------------+-------------------------------+ | MIN(QS1_MIN) | MIN(QS2_MIN) | MAX(QS3_MAX) | MAX(QS4_MAX) | (SUM(QS6_SUM)/SUM(QS5_COUNT)) | +-----------------+--------------+---------------+---------------+-------------------------------+ | 0.041714119635 | -6.1011707745 | 359.938579891 | 3.89870649736 | 112.537203939 | +----------------+---------------+---------------+---------------+-------------------------------+ {code}",4 DM-3394,"Cost Model Discovery","Learn about the cost model (LDM-144) and related documents in preparation for updating it per the contract",9 DM-3396,"Discourse evaluation (Part 2)","Work in support of evaluation Discourse as a DM platform for internal and external interactions.",7 DM-3397,"Find and evaluate multi-user password wallet for SQuaRE","Work to find and evaluate an off-the-shelf solution for sharing web services passwords between the SQuaRE group.",1 DM-3398,"Fix problem with default_engine","Fix the problem: {quote} 08/04/2015 05:39:47 werkzeug INFO: 141.142.237.30 - - [04/Aug/2015 17:39:47] ""GET /meta/v0/ HTTP/1.1"" 200 - 08/04/2015 05:39:49 __main__ ERROR: Exception on /meta/v0/db [GET] Traceback (most recent call last): File ""/home/becla/stack/Linux64/flask/0.10.1+8/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py"", line 1817, in wsgi_app response = self.full_dispatch_request() File ""/home/becla/stack/Linux64/flask/0.10.1+8/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py"", line 1477, in full_dispatch_request rv = self.handle_user_exception(e) File ""/home/becla/stack/Linux64/flask/0.10.1+8/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py"", line 1381, in handle_user_exception reraise(exc_type, exc_value, tb) File ""/home/becla/stack/Linux64/flask/0.10.1+8/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py"", line 1475, in full_dispatch_request rv = self.dispatch_request() File ""/home/becla/stack/Linux64/flask/0.10.1+8/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py"", line 1461, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File ""/nfs/home/becla/stack/repos/dax_metaserv/python/lsst/dax/metaserv/metaREST_v0.py"", line 59, in getDb return _resultsOf(text(query), scalar=True) File ""/nfs/home/becla/stack/repos/dax_metaserv/python/lsst/dax/metaserv/metaREST_v0.py"", line 122, in _resultsOf engine = current_app.config[""default_engine""] KeyError: 'default_engine' {quote}",1 DM-3399,"Research and Documenting the L1 System",NULL,4 DM-3400,"Eliminate circular aliases in slot centroid definition","[~smonkewitz] has discovered that our schema aliases for even the default configuration of measurement algorithms involve cycles, because the slot centroid algorithm contains a reference to its own flag. Fixing this should just involve an extra check in {{SafeCentroidExtractor}}.",1 DM-3401,"Explicitly disallow alias cycles in Schemas","The current guard against cycles is lazy and incomplete, as it seemed unlikely we'd ever have them. That's already been disproven (DM-3400), so it seems prudent to fix the guard code now.",6 DM-3404,"Port HSC updates to ingestImages.py","ingestImages.py provides a camera-agnostic manner of creating a data repository (including a registry). The HSC fork contains multiple improvements not present on the LSST side. We need these in order to ingest the HSC data.",2 DM-3409,"organize workspace functions discussion ",NULL,4 DM-3410,"workspace functions discussion",NULL,4 DM-3411,"workspace functions specification document","The first version of the document is here https://confluence.lsstcorp.org/pages/viewpage.action?pageId=41783931",20 DM-3412,"Review and edit import of Level 2 ICD milestones into DLP",NULL,2 DM-3414,"Clarify status of LSE-77","Work on LSE-77 _per se_ appears not to have kept up with the status of the substance of the interface requirements, as represented in, for instance, LSE-239. The action here is to see what change request actions may be appropriate for LSE-77 at this point.",1 DM-3415,"Review all DM ICDs for open issues and work with partner subsystems to clarify schedules for completion",NULL,4 DM-3416," workspace functions discussion",NULL,2 DM-3417,"Fix overestimation of aperture correction error","We're overestimating the aperture correction errors by including photon noise twice: both in the aperture correction errors and the original measurement errors. This is a migration of [HSC-1277|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1277] to LSST, which is where it should be fixed.",8 DM-3418,"Evaluate changes to LSE-209 and LSE-70","Action item from https://confluence.lsstcorp.org/display/SYSENG/2015+July+08-10+CCS-DAQ-OCS-DM+Workshop+IV Send initial feedback on LSE-70 and LSE-209",4 DM-3419,"obs_decam unit test for reading data ","The unit test wasn't working before and I edited the unit test of reading raw data. This got included with DM-3462. This unit test needs testdata_decam to be setup. The test fails with the stack b1597 at makeWcs (DM-3196). The afw branch u/yusra/DM-3196 is a temporary fix before DM-3196 is resolved. ",2 DM-3421,"Create testdata_decam ","Create a new testdata_decam repo with public instrument calibrated data. The 435MB file can be downloaded from http://uofi.box.com/testdata-decam",2 DM-3422,"S15 Qserv Release and Testing","This epic captures stories related to building, testing and maintaining Qserv releases, along with related documentation.",10 DM-3425," CI, Deploy and Distribution Improvements Part II","[Retitled epic to better capture current plan] This epic is an umbrella for Jenkins improvements, OpenStack and AWS automatic deployment, binary distribution, developer requests. Docker items in particular should go to a different epic. DM-2053 [JH 100%] ",49 DM-3426,"DLP/LDM-240 support chages - Part II","Service feature requests and bugfixes from JK, T/CAMs including format changes on sqre-jirakit.",2 DM-3427,"Meetings - Aug 2015 ","8/3: local group meeting 8/3: UIUC postdoc orientation 8/5: LSST2015 NCSA coordination meeting 8/10: local group meeting 8/17-22: LSST2015 Bremerton All Hands Meeting 8/24: local group meeting 8/28: Astro postdoc meeting",12 DM-3428,"Set up new desktop and install the stack","- Set up software on the new iMac - Installed the lsst stack with eups and lsstsw - Noticed a compiler requirement DM-3405 ",2 DM-3429,"Learn the development workflow and obs_decam status update ","- Learn git workflow and branching of the stack. - Had a long chat with Yusra about obs_decam - Reproduced obs_decam issues with different builds of the stack. There was a measurement failures that got solved with the newer builds of the stack. ",4 DM-3432,"Debug problem with timeout","If I run 4 simultaneous large queries: 3 object scans and 1 source scan, Xrootd silently died on 2 machines. Below I pasted the tail of the log files ccqserv108 0808 00:22:33.824 [0x7fb739583700] WARN Foreman (build/wdb/QueryAction.cc:109) - QueryAction overriding dbName with LSST 0808 00:22:33.824 [0x7fb74a00e700] INFO root (build/xrdsvc/ChannelStream.cc:122) - returning buffer (256, (more)) 0808 00:22:33.826 [0x7fb74a00e700] INFO root (build/xrdsvc/ChannelStream.cc:122) - returning buffer (62, (last)) 0808 00:22:33.826 [0x7fb74a00e700] INFO root (build/xrdsvc/SsiSession.cc:153) - RequestFinished type=isStream 0808 00:22:34.468 [0x7fb739583700] INFO root (build/wdb/QueryAction.cc:261) - &&& _fillRows size=8 0808 00:22:34.469 [0x7fb739583700] DEBUG root (build/wdb/QueryAction.cc:288) - _transmit last=1 0808 00:22:34.469 [0x7fb739583700] DEBUG root (build/wdb/QueryAction.cc:307) - _transmitHeader 0808 00:22:34.469 [0x7fb739583700] INFO root (build/proto/ProtoHeaderWrap.cc:52) - msgBuf size=256 -> [[0]=40, [1]=13, [2]=2, [3]=0, [4]=0, ..., [251]=48, [252]=48, [253]=48, [254]=48, [255]=48] 0808 00:22:34.469 [0x7fb739583700] INFO root (build/xrdsvc/SsiSession_ReplyChannel.cc:85) - sendStream, checking stream 0 len=256 last=0 ----- ccqserv124 0808 00:22:25.329 [0x7f25a67bf700] DEBUG ScanSched (build/wsched/ChunkDisk.cc:234) - ChunkDisk registering for 2044 : SELECT MIN(ra) AS QS1_MIN,MAX(ra) AS QS2_MAX,MIN(decl) AS QS3_MIN,MAX(decl) AS QS4_MAX FROM LSST.Object_2044 AS QST_1_ p=0x7f257003d0b8 0808 00:22:25.329 [0x7f25a67bf700] INFO Foreman (build/wcontrol/Foreman.cc:296) - Runner running Task: msg: session=7 chunk=2044 db=LSST entry time=Fri Aug 7 23:52:06 2015 frag: q=SELECT MIN(ra) AS QS1_MIN,MAX(ra) AS QS2_MAX,MIN(decl) AS QS3_MIN,MAX(decl) AS QS4_MAX FROM LSST.Object_2044 AS QST_1_, sc= rt=r_7bff268d0e369dda8fa314132538a96ad_2044_0 0808 00:22:25.329 [0x7f25a67bf700] INFO Foreman (build/wdb/QueryAction.cc:177) - Exec in flight for Db = q_b762c96f418726ae3457c74c0350d0c4 0808 00:22:25.329 [0x7f25a67bf700] WARN Foreman (build/wdb/QueryAction.cc:109) - QueryAction overriding dbName with LSST 0808 00:22:25.330 [0x7f25a50b6700] INFO root (build/xrdsvc/ChannelStream.cc:122) - returning buffer (44, (last)) 0808 00:22:25.330 [0x7f25a50b6700] INFO root (build/xrdsvc/SsiSession.cc:153) - RequestFinished type=isStream 0808 00:22:26.218 [0x7f25a67bf700] INFO root (build/wdb/QueryAction.cc:261) - &&& _fillRows size=81 0808 00:22:26.218 [0x7f25a67bf700] DEBUG root (build/wdb/QueryAction.cc:288) - _transmit last=1 0808 00:22:26.218 [0x7f25a67bf700] DEBUG root (build/wdb/QueryAction.cc:307) - _transmitHeader 0808 00:22:26.218 [0x7f25a67bf700] INFO root (build/proto/ProtoHeaderWrap.cc:52) - msgBuf size=256 -> [[0]=40, [1]=13, [2]=2, [3]=0, [4]=0, ..., [251]=48, [252]=48, [253]=48, [254]=48, [255]=48] 0808 00:22:26.218 [0x7f25a67bf700] INFO root (build/xrdsvc/SsiSession_ReplyChannel.cc:85) - sendStream, checking stream 0 len=256 last=0 ",4 DM-3433,"Fix socket timeout problem in xrootd framework","When we run a query that take long time, client times out, it closes the socket, which triggers cancellation on the server side.",4 DM-3435,"W16 End-to-End Integration (Data Access portion)","End to end system integration. This epic covers work related to # loading data generated by pipelines into Qserv # fully integrating Qserv with SUI",22 DM-3436,"Qserv - webserv integration","Setup Qserv and configure webserv to talk to Qserv. Verify all works, and fix discovered problems.",2 DM-3437,"Add column names metadata to db query results","Per discussion at data access meeting Aug 10, it'd be good to send column names with the query results.",2 DM-3438,"Revisit KPIs for Qserv","Need to come up with KPIs for Qserv ",2 DM-3439,"Rename ingest.py to reduce confusion with database","The {{ingestImages.py}} bin script provides a camera-agnostic manner of creating a data repository (including a registry). The back-end code resides in pipe_tasks under the name {{ingest.py}}, and the {{IngestTask._DefaultName = ""ingest""}}, which means that configuration files in obs packages are also named {{ingest.py}}. This choice of name was unfortunate, as it may be confused with ingest of sources into the database. We should change the name to reduce this confusion, perhaps {{ingestImages.py}} like the bin script.",2 DM-3440,"add meas_extensions_photometryKron to lsstsw, lsst_distrib","meas_extensions_photometryKron should be added to the CI system, since we are trying to keep it updated. This is blocked by DM-2429 because that includes a fix for a unit test (which the CI system would have caught).",1 DM-3442,"Processing y-band HSC data fails in loading reference sources","{code} processCcd.py /lsst3/HSC/data/ --output /raid/price/test --id visit=904400 ccd=50 [...] processCcd.calibrate.astrometry.solver.loadAN: Loading reference objects using center (1023.5, 2091) pix = Fk5Coord(319.8934727, -0.0006943, 2000.00) sky and radius 0.111920792477 deg processCcd FATAL: Failed on dataId={'taiObs': '2013-11-03', 'pointing': 672, 'visit': 904400, 'dateObs': '2013-11-03', 'filter': 'HSC-Y', 'field': 'STRIPE82L', 'ccd': 50, 'expTime': 30.0}: Could not find flux field(s) y_camFlux, y_flux Traceback (most recent call last): File ""/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/cmdLineTask.py"", line 320, in __call__ result = task.run(dataRef, **kwargs) File ""/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py"", line 118, in wrapper res = func(self, *args, **keyArgs) File ""/home/lsstsw/stack/Linux64/pipe_tasks/10.1-28-gf9582e4+2/python/lsst/pipe/tasks/processCcd.py"", line 85, in run result = self.process(sensorRef, postIsrExposure) File ""/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py"", line 118, in wrapper res = func(self, *args, **keyArgs) File ""/home/lsstsw/stack/Linux64/pipe_tasks/10.1-28-gf9582e4+2/python/lsst/pipe/tasks/processImage.py"", line 160, in process calib = self.calibrate.run(inputExposure, idFactory=idFactory) File ""/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py"", line 118, in wrapper res = func(self, *args, **keyArgs) File ""/home/lsstsw/stack/Linux64/pipe_tasks/10.1-28-gf9582e4+2/python/lsst/pipe/tasks/calibrate.py"", line 457, in run astromRet = self.astrometry.run(exposure, sources1) File ""/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py"", line 118, in wrapper res = func(self, *args, **keyArgs) File ""/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetAstrometry.py"", line 177, in run results = self.astrometry(sourceCat=sourceCat, exposure=exposure, bbox=bbox) File ""/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py"", line 118, in wrapper res = func(self, *args, **keyArgs) File ""/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetAstrometry.py"", line 292, in astrometry astrom = self.solver.determineWcs(sourceCat=sourceCat, exposure=exposure, bbox=bbox) File ""/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetBasicAstrometry.py"", line 409, in determineWcs return self.determineWcs2(sourceCat=sourceCat, **margs) File ""/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetBasicAstrometry.py"", line 437, in determineWcs2 astrom = self.useKnownWcs(sourceCat, wcs=wcs, **kw) File ""/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetBasicAstrometry.py"", line 308, in useKnownWcs calib = None, File ""/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py"", line 118, in wrapper res = func(self, *args, **keyArgs) File ""/home/lsstsw/stack/Linux64/meas_algorithms/10.1-15-g0d3ecf6/python/lsst/meas/algorithms/loadReferenceObjects.py"", line 173, in loadPixelBox loadRes = self.loadSkyCircle(ctrCoord, maxRadius, filterName) File ""/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py"", line 118, in wrapper res = func(self, *args, **keyArgs) File ""/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/loadAstrometryNetObjects.py"", line 141, in loadSkyCircle fluxField = getRefFluxField(schema=refCat.schema, filterName=filterName) File ""/home/lsstsw/stack/Linux64/meas_algorithms/10.1-15-g0d3ecf6/python/lsst/meas/algorithms/loadReferenceObjects.py"", line 40, in getRefFluxField raise RuntimeError(""Could not find flux field(s) %s"" % ("", "".join(fluxFieldList))) RuntimeError: Could not find flux field(s) y_camFlux, y_flux {code} We should be able to fix this by setting config parameters (e.g., {{calibrate.astrometry.solver.defaultFilter}} or {{calibrate.astrometry.solver.filterMap}}), but how do we keep that synched with the choice of reference catalog? And once we get past astrometry, we also have the same problem in photocal.",2 DM-3444,"Add support for parsing user log files","In order to get timings for jobs executed, add support for non-dagman generated log files. These are the user log files that HTCondor writes out for each individual job.",6 DM-3447,"Improve packaging of shared libraries in scons","As discovered through DM-3161, our swig-generated libraries are messy. Specifically, we are dumping everything we might need into the czarLib library. That includes mysql and mysql-client related things. CssLib needs mysql functions too. Given that czar imports both these libraries, we ended up with duplicate symbols. That is being patched in DM-3161, but it needs a further look / cleanup. We need to break things into smaller libraries. Difficulty: understanding dependencies and avoiding circular dependencies.",12 DM-3448,"S17 Long-running Query Optimizations","As discovered through DM-3432 when a query runs for long time, czar does not get response from worker for a long time, and times out. A quick patch we did during Summer 2015 tests was to increase the timeout to a large number. The issue was discussed at [Qserv mtg Aug 12 2015|https://confluence.lsstcorp.org/display/DM/Database+Meeting+2015-08-12]. If we knew how long a query was going to take, we could set the timeout to appropriate value, but we can't always estimate well how long the query is going to take. The best solution we came up with: periodically check with the worker asynchronously what the status of the query is. Worker should respond with something like: queued, scheduled, working / started x sec ago. Note, this might require changes to the xrdssi API.",26 DM-3449,"Handle problems with connecting to mysql in czar","We observed in S15 tests that if we run too many queries, czar is running out of connections to mysql and as a result through exception that is uncaught (and dies). We triggered this by starting 110 queries. To ""fix"" this problem we increased the max_connections in etc/my.cnf from 256 to 512. So, most likely the easiest way to reproduce it would be to set the max_connections to a very small number. This story involves handling the ""uncaught exception"" gracefully.",3 DM-3450,"Tweaks to configurations discovered during S15 tests","Apply tweaks we found useful when running large scale tests. This includes: # etc/my.cnf: change max_connections to 512 # add"": {quote}export XRD_REQUESTTIMEOUT=64000 export XRD_STREAMTIMEOUT=64000 export XRD_DATASERVERTTL=64000 export XRD_TIMEOUTRESOLUTION=64000{quote} to init.d/qserv-czar # add : {quote}ulimit -c unlimited{quote} to all startup scripts in init.d. This will make sure core file is always dumped when we have problems.",1 DM-3451,"Resolve problem with running many simultaneous queries","When we run with 110 simultaneous queries, czar fails with ""uncaught exception""",2 DM-3452,"Integrate pipelines with MySQL and Qserv","Load data produced by pipelines into MySQL (on lsst10), and Qserv",5 DM-3453,"AstrometryTask.run return not consistent with ANetAstrometryTask","ANetAstrometryTask.run returns matchMetadata but AstrometryTask.run returns matchMeta. The two must agree. It turns out that matchMeta is more widely used, so I'll standardize on that.",1 DM-3455,"ProcessImageTask.matchSources fails if using ANetAstrometryTask","ProcessImageTask.matchSources fails when using ANetAstrometryTask with the following error: {code} processCcd.calibrate.astrometry: Applying distortion correction processCcd FATAL: Failed on dataId={'taiObs': '2013-11-03', 'pointing': 672, 'visit': 904400, 'dateObs': '2013-11-03', 'filter': 'HSC-Y', 'field': 'STRIPE82L', 'ccd': 50, 'expTime': 30.0}: File ""src/table/Schema.cc"", line 239, in lsst::afw::table::SchemaItem lsst::afw::table::detail::SchemaImpl::find(const string&) const [with T = double; std::string = std::basic_string] Field or subfield withname 'astrom_distorted_x' not found with type 'D'. {0} lsst::pex::exceptions::NotFoundError: 'Field or subfield withname 'astrom_distorted_x' not found with type 'D'.' Traceback (most recent call last): File ""/ssd/rowen/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+3/python/lsst/pipe/base/cmdLineTask.py"", line 320, in __call__ result = task.run(dataRef, **kwargs) File ""/ssd/rowen/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+3/python/lsst/pipe/base/timer.py"", line 118, in wrapper res = func(self, *args, **keyArgs) File ""/ssd/rowen/lsstsw/stack/Linux64/pipe_tasks/tickets.DM-3453-g086c9ddd0a/python/lsst/pipe/tasks/processCcd.py"", line 85, in run result = self.process(sensorRef, postIsrExposure) File ""/ssd/rowen/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+3/python/lsst/pipe/base/timer.py"", line 118, in wrapper res = func(self, *args, **keyArgs) File ""/ssd/rowen/lsstsw/stack/Linux64/pipe_tasks/tickets.DM-3453-g086c9ddd0a/python/lsst/pipe/tasks/processImage.py"", line 219, in process srcMatches, srcMatchMeta = self.matchSources(calExposure, sources) File ""/ssd/rowen/lsstsw/stack/Linux64/pipe_tasks/tickets.DM-3453-g086c9ddd0a/python/lsst/pipe/tasks/processImage.py"", line 250, in matchSources astromRet = astrometry.loadAndMatch(exposure=exposure, sourceCat=sources) File ""/ssd/rowen/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+3/python/lsst/pipe/base/timer.py"", line 118, in wrapper res = func(self, *args, **keyArgs) File ""/ssd/rowen/lsstsw/stack/Linux64/meas_astrom/tickets.DM-3453-gbb2ad1f49c/python/lsst/meas/astrom/anetAstrometry.py"", line 321, in loadAndMatch with self.distortionContext(sourceCat=sourceCat, exposure=exposure) as bbox: File ""/ssd/rowen/lsstsw/anaconda/lib/python2.7/contextlib.py"", line 17, in __enter__ return self.gen.next() File ""/ssd/rowen/lsstsw/stack/Linux64/meas_astrom/tickets.DM-3453-gbb2ad1f49c/python/lsst/meas/astrom/anetAstrometry.py"", line 295, in distortionContext sourceCat.table.defineCentroid(self.distortedName) File ""/ssd/rowen/lsstsw/stack/Linux64/afw/10.1-37-gaedf466/python/lsst/afw/table/tableLib.py"", line 8887, in defineCentroid return _tableLib.SourceTable_defineCentroid(self, *args) NotFoundError: File ""src/table/Schema.cc"", line 239, in lsst::afw::table::SchemaItem lsst::afw::table::detail::SchemaImpl::find(const string&) const [with T = double; std::string = std::basic_string] Field or subfield withname 'astrom_distorted_x' not found with type 'D'. {0} lsst::pex::exceptions::NotFoundError: 'Field or subfield withname 'astrom_distorted_x' not found with type 'D'.' {code} This is probably a result of DM-2939. The basic problem is that the distortion context in ANetAstrometryTask should not be run at that point in processing. [~price] suggests that a simple clean fix is to make the distortion context a no-op if the WCS already contains distortion, if that works. This is what I will try first.",1 DM-3456,"Fix problems with talking from webserv to qserv","Flask or sqlalchemy which are part of webserv are producing some extra queries that are confusing qserv. So basically, at the moment even the simplest query run via webserv that is directed to qserv fails.",2 DM-3457,"Implement prototype stack documentation with Sphinx","Implement a minimally-viable Sphinx documentation repository for the LSST stack. The code is available at https://github.com/lsst-sqre/lsst_stack_docs",5 DM-3458,"Research existing sphinx doc implementations","Examine how python packages such as astropy structure and implement their sphinx docs.",2 DM-3459,"make forced and SFM interfaces more consistent","From [~rowen]: {quote} SimpleMeasurementTask.run and ForcedMeasurementTask.run now both take a source catalog, but the two use the opposite order for the first two arguments (one has the catalog first, the other has the exposure first) {quote}",1 DM-3460,"applyApCorr mis-handles missing data","In ApplyApCorrTask.run the following lines do not behave as expected because get returns None if the data is missing, rather than raising an exception: {code} try: apCorrModel = apCorrMap.get(apCorrInfo.fluxName) apCorrSigmaModel = apCorrMap.get(apCorrInfo.fluxSigmaName) except Exception: {code}",1 DM-3462,"Make obs_decam handle raw data ","The current obs_decam expects instrument calibrated data from the community pipeline, i.e. it requires matching instcal (Instrument Calibrated), dqmask (the associated mask file), and wtmap (weight map) data from the same visit. This issue is to add functionality so that raw DECam images can be ingested into registry and retrieved by the data butler. Practically, this will create new or expand existing sub-classes of CameraMapper and IngestTask. A brief summary of changes: - The unit test getRaw.py is updated and should pass, with DM-3196 - Working testdata_decam for the unit test is currently at lsst-dev /lsst8/testdata_decam and https://uofi.box.com/testdata-decam - DecamInstcalMapper is renamed to DecamMapper, to reflect that Butler can also get ""raw"" now besides ""instcal"". Please update _mapper in your data repositories. - To create a registry for raw data, run {code:java} ingestImagesDecam.py /path/to/repo --mode=link --filetype=""raw"" /path/*.fits.fz {code} - The default filetype is ""instcal"" for ingestImagesDecam.py, so previous use for instcal stays. ",13 DM-3463,"psfex lapack symbols may collide with built in lapack","On my Mac meas_extensions_psfex fails to build due to the numpy config test failing. ""import numpy"" fails with: {code} dlopen(/Users/rowen/LSST/lsstsw/anaconda/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so, 2): can't resolve symbol __NSConcreteStackBlock in /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib because dependent dylib #1 could not be loaded in /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib {code} Our best guess (see discussion in Data Management 2015-08-14 at approx. 1:57 PM Pacific time) is that the special lapack functions in psfex are colliding with the lapack that anaconda uses. In case it helps I see this on OS X 10.9.5. I do not see it on lsst-dev.",2 DM-3464,"lsstswBuild.sh --print-fail should report config.log",NULL,1 DM-3467,"Make async cancellation more flexible",NULL,8 DM-3468,"drawing text to ds9 fails if size or the font family is set","Commands like {code} ds9.dot('xxxx', 100, 100, size=3) ds9.dot('xxxx', 100, 120, fontFamily=""times"") {code} silently fail. The problem is that commands like {code} xpaset -p ds9 regions command '{text 100 100 # text=xxxx color=red font=""times 12""}' {code} fail; you need to say {{font=times 12 normal}}",1 DM-3469,"Prepare for S15 end-to-end system and Firefly relesase","For S15, DM will try to have an end-to-end system in place, and work out all the issues during the process. SUI will try to a web application to access some of dada access APIs (dax-) on a system at NCSA. SUI will have a simple Firefly release to the community.",15 DM-3470,"Install/deploy SUI web application at NCSA","For summer 15 release, we will deploya SUI web app on NCSA accessible to DM team. - work with NCSA to have a server setup - install necessary software packages - install SUI software - deploy the system and test ",5 DM-3471,"A simple Firefly release for S15","TO install and deploy a Firefly application from scratch will take a few hours, including getting all the necessary software packages and install them all. In order for users to get the system up and running in as little time as possible, we will provide a war file with embedded tomcat server. Users don;t need any third party software except Java1.8. The time to get the system up and running after installing Java1.8 will be a minute after the download of war file.",10 DM-3472,"Implement Basic spatial lookups for the Butler ","Based on [this |https://confluence.lsstcorp.org/display/DM/Dependencies+on+02C.06+Science+Data+Archive+and+Application+Services] pipeline software will require spatial lookups via butler. Current plan is: Provide an ""ingest"" script that looks at a butler exposure dataset, pulls spatial information and data ID out of it, then shoves all of that into an sqlite3 table that can be used for accelerated spatial lookups. Then provide a task that takes skymap parameters, and maps those to potentially overlapping exposures: the sqlite3 table is outside butler; and the will use a spatial location to get a dataId, and then pass that dataId into butler to get that file. ",10 DM-3474,"Debug problems with near neighbor queries","The following near neighbor query: {quote} select o1.ra as ra1, o2.ra as ra2, o1.decl as decl1, o2.decl as decl2, scisql_angSep(o1.ra, o1.decl,o2.ra, o2.decl) AS theDistance from Object o1, Object o2 where qserv_areaspec_box(90.299197, -66.468216, 98.762526, -56.412851) and scisql_angSep(o1.ra, o1.decl, o2.ra, o2.decl) < 0.015 {quote} fails on the IN2P3 cluster.",4 DM-3477,"Design SQL API for getting query type","When unsure, user should be able to check what type of query a given query is (for example, is the query ""select * from Object where qserv_area_spec(1, 2, 10, 5)"" considered interactive or async? This story involves deciding how the SQL API will look like.",2 DM-3478,"Design API for passing query type","User should be able to pass hint with a query indicating what query type it is. Based on that the return result will either be the query result, or queryId. This story involves designing the API (sql and RESTful).",3 DM-3479,"Design RESTful APIs for async queries in WebServ","Need RESTul API for: * retrieving partial results while query is running * killing async queries",2 DM-3480,"Design SQL APIs for async queries","Need SQL API for: * submitting async query, note that we should be able to specify where the results are going / what is the format of the results * retrieving status of async query * retrieving results of async query * retrieving partial results of async query while it is running ",5 DM-3481,"adapt sandbox-jenkins-demo to changes in jfryman/nginx 0.2.7","Changes in the way jfryman/nginx 0.2.7 handles tls cert files since 0.2.6 have run awful of selinux permissions issues.",2 DM-3482,"Attending Cyber Security Summit","Attending NSF Cyber Security Summit in my capacity as LSST ISO.",3 DM-3483,"Calibration transformation should not fail on negative flux","Before database ingest, measured source fluxes are converted to magnitudes as per DM-2305. The default behaviour of {{afw::image::Calib}} is to throw when a negative flux is encountered, which derails the whole transformation procedure. Better is to return a NaN.",1 DM-3484,"Design RESTful API for getting query type","When unsure, user should be able to check what type of query a given query is (for example, is the query ""select * from Object where qserv_area_spec(1, 2, 10, 5)"" considered interactive or async? This story involves deciding how the RESTful API will look like.",2 DM-3488,"Debug problem with large results set","Query returning 2 billion rows causes problems for czar - czar is using nearly 16 GB or memory. Need to understand why RAM usage in czar is correlated with result size.",1 DM-3489,"replace Calib negative flux behavior methods with context manager","DM-3483 adds a nice context manager for handling the behavior of Calib objects when encountering negative fluxes. This could be even nicer if we moved it into afw and integrated it with Calib itself, replacing the existing static methods (which are bad because they use global variables). This will be an API change, and will require an RFC.",2 DM-3490,"Quick-and-dirty n-way spatial matching","This issue will add limited N-way spatial matching of multiple catalogs with identical schemas, sufficient for measuring FY15 KPMs. It will be a simple wrapper on our existing 2-way matching code in afw, and will not be intended for long term use (as it won't be an efficient algorithm or an ideal interrface).",2 DM-3491,"Update ip_diffim to use the new NO_DATA flag instead of EDGE","In ip_diffImm some uses of EDGE were converted to or supplemented with NO_DATA, but others were not. This ticket handles the missing instances.",1 DM-3492,"Correct for distortion in matchOptimisticB astrometry matcher","matchOptimisticB does not correct for distortion, although an estimate of the distortion is available. We suspect that doing the matching on the celestial sphere might be ideal, but matching on a tangent plane has worked for HSC.",5 DM-3493,"Fix crosstalk following ds9 interface changes","crosstalk.py in obs_subaru uses ds9 without actually displaying anything, which causes trouble if display_ds9 is not setup.",1 DM-3494,"lsst.afw.display.setMaskTransparency doc doesn't match code","The docstring for {{setMaskTransparency}} says ""Specify display's mask transparency (percent); or *None to not set it when loading masks*"", but (from inspection of the code), it doesn't do anything if {{transparency}} is {{None}}. ",1 DM-3495,"X16 Large Results","As described in [LDM-135|http://ldm-135.readthedocs.org/en/master/#query-access-related-requirements] high volume queries can return large results (~6 GB per query). Current version of Qserv poorly supports large results. We have observed that a single query that returns ~40 million or two-column rows (~0.5 GB) requires nearly 16 GB of RAM in czar, and czar uses 100% of CPU for extended period of time. This is most likely related to aggregating results - results are currently cached in memory before they are returned to the proxy. This epic involves redesigning this part of Qserv. One reasonable approach would be to write to separate tables from each worker, and deliver the final result as UNION of these tables. Or, perhaps we could use a shared files system and each worker would write results directly there, without burdening czar. Deliverable: demonstration involving 8 queries each returning 2 GB result handled through a single czar. Czar should not use excessive amount of RAM or CPU.",19 DM-3496,"Add support for images produced by pipelines in end-to-end integration test","We need to add support for images that are produced by pipelines which are run as part of the end-to-end integration tests - ImgServ should be able to serve these images.",6 DM-3499,"Add support for executing async queries through czar",NULL,14 DM-3500,"S17 Async Queries in WebServ","Add support for async queries in webserv. Deliverable: webserv that allows to manage async queries (start, kill, retrieve results). Partial results are not covered here.",12 DM-3501,"Implement RESTful API for async queries in WebServ",NULL,10 DM-3502,"Research mysql proxy alternatives","Research alternatives to mysql proxy, including things like maxScale.",9 DM-3504,"Improve spatial image search for butler",NULL,11 DM-3505,"Modify czar to use per query CSS metadata","Czar is currently caching CSS information, the snapshot is taken when czar starts. Once we start dealing with more dynamic system where databases and tables can come and go anytime while the system is up (in particular, L3 databases and tables), the metadata in CSS would need to be refreshed per query to stay up to date",6 DM-3506,"W16 Support Dynamic CSS Metadata in Czar","Czar needs to support dynamic CSS metadata. This epic involves reworking Facade and related code so that czar can have up to date CSS metadata (per query) instead of relying on static snapshots",40 DM-3508,"Revisit Facade API",NULL,6 DM-3511,"Adopt Webserv to work with reworked db module",NULL,4 DM-3512,"Expose column metadata via metaserv","We need to expose via RESTful APIs information about columns such as units, ucds, column description etc. It will require querying information_schema with information from our DDT tables. This should include exposing information which columns are the ra/decl columns used for indexing.",6 DM-3513,"Add support for row counts in Metaserv","SUI software would like to frequently check what the row counts for our tables. This story involves caching the information about the row count information in the metaserv. Note that for L3 tables, that might change, we will need to intercept queries that alter these tables, and update the cache.This is beyond the scope of this story.",4 DM-3514,"Add flag to MetaServ showing if qserv_area_spec is available","SUI software needs to know which tables support spatial queries. This story involves exposing this information via Metaserv (the information is available via CSS) ",3 DM-3515,"Refactor DipoleMeasurement: Dipole classification to plugin","DipoleMeasurementTask currently runs dipole measurement plugins and runs its own implemented dipole classification method. This ticket translates dipole classification to a plugin itself to simplify DipoleMeasurementTask. Instead of inheriting from SingleFrameMeasurementTask, DipoleMeasurementTask will run SingleFrameMeasurementTask setting the appropriate default slots and plugins (including dipole classification plugin). ",6 DM-3516,"Base DC network design","Design proposal for Base DC network",20 DM-3518,"prune stale obs_subaru dependencies","obs_subaru has some Eups dependencies that should be removed: - meas_extensions_multiShapelet (removed from the LSST stack, content moved to meas_modelfit) - meas_multifit (renamed to meas_modelfit) - fitsthumb (need to cherry-pick code from HSC to replace it) - pyfits (investigate what we use it for; it might not be needed) - psycopg2 (I don't think we'll ever use this on the LSST side; we should add it back to the HSC side after we re-fork)",4 DM-3519,"Create flux environment for input panels",NULL,20 DM-3520,"Organize React component to use new flux environment",NULL,10 DM-3522,"Releasing un-acquired resources bug","Running a mix of queries: 75 low volume and 10 high volume that include near neighbor failed at some point with {quote} terminate called after throwing an instance of 'lsst::qserv::Bug' what(): ChunkResource ChunkEntry::release: Error releasing un-acquired resource {quote} Stack trace: {quote} #0 0x00007fb6b0cce5e9 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install expat-2.1.0-8.el7.x86_64 glibc-2.17-78.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.12.2-14.el7.x86_64 libcom_err-1.42.9-7.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 libicu-50.1.2-11.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libstdc++-4.8.3-9.el7.x86_64 nss-softokn-freebl-3.16.2.3-9.el7.x86_64 openssl-libs-1.0.1e-42.el7_1.9.x86_64 pcre-8.32-14.el7.x86_64 xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64 (gdb) where #0 0x00007fb6b0cce5e9 in raise () from /lib64/libc.so.6 #1 0x00007fb6b0ccfcf8 in abort () from /lib64/libc.so.6 #2 0x00007fb6b15d29b5 in __gnu_cxx::__verbose_terminate_handler() () from /lib64/libstdc++.so.6 #3 0x00007fb6b15d0926 in ?? () from /lib64/libstdc++.so.6 #4 0x00007fb6b15cf8e9 in ?? () from /lib64/libstdc++.so.6 #5 0x00007fb6b15d0554 in __gxx_personality_v0 () from /lib64/libstdc++.so.6 #6 0x00007fb6b1069913 in ?? () from /lib64/libgcc_s.so.1 #7 0x00007fb6b1069e47 in _Unwind_Resume () from /lib64/libgcc_s.so.1 #8 0x00007fb6ab554247 in lsst::qserv::wdb::ChunkResourceMgr::Impl::release (this=0x21d1cc0, i=...) at build/wdb/ChunkResource.cc:398 #9 0x00007fb6ab552696 in lsst::qserv::wdb::ChunkResource::~ChunkResource (this=0x7fb68a5f9b70, __in_chrg=) at build/wdb/ChunkResource.cc:131 #10 0x00007fb6ab560f0f in lsst::qserv::wdb::QueryAction::Impl::_dispatchChannel (this=0x7fb65848c4d0) at build/wdb/QueryAction.cc:392 #11 0x00007fb6ab55f5ab in lsst::qserv::wdb::QueryAction::Impl::act (this=0x7fb65848c4d0) at build/wdb/QueryAction.cc:187 #12 0x00007fb6ab562084 in lsst::qserv::wdb::QueryAction::operator() (this=0x7fb658050548) at build/wdb/QueryAction.cc:450 #13 0x00007fb6ab544f46 in lsst::qserv::wcontrol::ForemanImpl::Runner::operator() (this=0x7fb67400fa20) at build/wcontrol/Foreman.cc:302 #14 0x00007fb6ab551cf0 in std::_Bind_simple::_M_invoke<>(std::_Index_tuple<>) ( this=0x7fb67400fa20) at /usr/include/c++/4.8.2/functional:1732 #15 0x00007fb6ab551a8b in std::_Bind_simple::operator()() (this=0x7fb67400fa20) at /usr/include/c++/4.8.2/functional:1720 {quote} Tail of log file from xrootd log: {quote} 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG GroupSched (build/wsched/GroupScheduler.cc:139) - _getNextTasks(1)>->-> 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG GroupSched (build/wsched/GroupScheduler.cc:151) - Returning 1 to launch 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG GroupSched (build/wsched/GroupScheduler.cc:154) - _getNextTasks <<<<< 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG ScanSched (build/wsched/ScanScheduler.cc:172) - _getNextTasks(29)>->-> 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG ScanSched (build/wsched/ChunkDisk.cc:199) - ChunkDisk busyness: yes 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG ScanSched (build/wsched/ChunkDisk.cc:171) - ChunkDisk getNext: current= (scan=10436, cached=8360,8259,) candidate=10301 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG ScanSched (build/wsched/ChunkDisk.cc:184) - ChunkDisk denying task 0821 19:08:58.530 [0x7fb68a6fb700] DEBUG ScanSched (build/wsched/ScanScheduler.cc:196) - _getNextTasks <<<<< 0821 19:08:58.531 [0x7fb68a6fb700] INFO root (build/xrdsvc/SsiSession.cc:120) - Enqueued TaskMsg for Resource(/chk/LSST/2732) in 0.001016 seconds 0821 19:08:58.531 [0x7fb6895f8700] DEBUG Foreman (build/wcontrol/Foreman.cc:175) - Registered runner 0x7fb66c141ab0 0821 19:08:58.531 [0x7fb6895f8700] DEBUG Foreman (build/wcontrol/Foreman.cc:209) - Started task Task: msg: session=434445 chunk=2732 db=LSST entry time= frag: q=SELECT o.deepSourceId,o.ra,o.decl,s.coord_ra,s.coord_decl,s.parent FROM LSST.Object_2732 AS o,LSST.Source_2732 AS s WHERE scisql_s2PtInBox(o.ra,o.decl,48.482655,-54.274507,48.555903,-54.196952)=1 AND scisql_s2PtInBox(s.coord_ra,s.coord_decl,48.482655,-54.274507,48.555903,-54.196952)=1 AND o.deepSourceId=s.objectId, sc= rt=r_4344458c9456ede5cbe0b5f42e1a1571d5dd73_2732_0 0821 19:08:58.531 [0x7fb6895f8700] INFO Foreman (build/wcontrol/Foreman.cc:296) - Runner running Task: msg: session=434445 chunk=2732 db=LSST entry time= frag: q=SELECT o.deepSourceId,o.ra,o.decl,s.coord_ra,s.coord_decl,s.parent FROM LSST.Object_2732 AS o,LSST.Source_2732 AS s WHERE scisql_s2PtInBox(o.ra,o.decl,48.482655,-54.274507,48.555903,-54.196952)=1 AND scisql_s2PtInBox(s.coord_ra,s.coord_decl,48.482655,-54.274507,48.555903,-54.196952)=1 AND o.deepSourceId=s.objectId, sc= rt=r_4344458c9456ede5cbe0b5f42e1a1571d5dd73_2732_0 0821 19:08:58.531 [0x7fb6895f8700] INFO Foreman (build/wdb/QueryAction.cc:177) - Exec in flight for Db = q_fd51ad249f62fb765e173d7b3cae5d94 0821 19:08:58.531 [0x7fb6895f8700] WARN Foreman (build/wdb/QueryAction.cc:109) - QueryAction overriding dbName with LSST 0821 19:08:58.718 [0x7fb6a0d8e700] INFO root (build/wdb/QueryAction.cc:261) - &&& _fillRows size=106 0821 19:08:58.718 [0x7fb6a0d8e700] INFO root (build/wdb/QueryAction.cc:261) - &&& _fillRows size=210 0821 19:08:58.718 [0x7fb6a0d8e700] INFO root (build/wdb/QueryAction.cc:261) - &&& _fillRows size=316 ...(thousands of _fillRows lines) terminate called after throwing an instance of 'lsst::qserv::Bug' what(): ChunkResource ChunkEntry::release: Error releasing un-acquired resource {quote} ",3 DM-3523,"data center requirements",NULL,2 DM-3524,"Revise LDM-240","Revising LDM-240.",10 DM-3525,"Add ITIL model use cases to Enterprise Architect",NULL,2 DM-3526,"Prepare use case diagrams for LSST 2015 Operations Concepts breakout session",NULL,2 DM-3527,"FY19 Query Result Caching","Implement Query caching. Note that most likely we can't ""just"" rely on mysql caching, and we will need to do some custom tweaks, in particular for async queries. I'd be useful to allow users to ""pin"" results from interesting queries. Typical usecase: user runs a query (it can be interactive) and then decides to keep the results for longer time. ",79 DM-3530,"LOE - Week ending 8/21/15",NULL,3 DM-3531,"LOE - Week ending 8/28/15",NULL,4 DM-3533,"Information categorization for docushare","Project started to categorize documents on docushare as per LSST's information categorization policy.",3 DM-3534,"Meet with Scott K for IdM project","Scott K will come to NCSA to discuss initial IdM requirements for LSST.",1 DM-3535,"NIST SP 800.82 investigation","NIST SP 800.82 might contain useful information for securing the scada enclave at the observatory site. Or it might even be the model we should use in full.",2 DM-3536,"Czar Failover in xrdssi","In production we will need to recover from czar failures by automatically failing over to a different czar. This epic involves designing and implementing the interfaces in xrootd xrdssi that will be need to support czar fail over.",20 DM-3538,"Translate more mask pixel bits from instcal data quality mask of DECam data ","The mask pixel bits are defined differently in DECam instcal data from different pipelines, see http://community.lsst.org/t/decam-data-quality-masks/133 for a summary. The current mapping in DecamInstcalMapper is for the community pipeline Pre-V3.5.0. More mask bits were defined in DESDM y1 products. This issue provides full mapping for DESDM y1 products so all questionable pixels are translated into the MaskedImage. ",2 DM-3539,"Fix bugs found in FitsRead ","There was a bug found in writeFitsFile(OutputStream stream, FitsRead[] fitsReadAry, Fits refFits) method where it saved the data from refFits into the output Fits file instead of the data from the FitsRead object. There is another possible bug under investigation. The output Fits file created by FitsRead.createNewFits() can not be opened as an image. ",4 DM-3540,"Refactor ChunkResource for testability","While the wdb/testChunkResource.cc unit test appears to exercise the chunk resource management code, it does not seem to actually test it. We should refactor the classes in the ChunkResource header and implementation files to make the sanity of the implementation checkable. In particular, I think that Backend from ChunkResource.cc should be a public interface (with a less generic name) that the unit test can implement, and that ChunkResourceMgr should be a concrete class implemented in terms of a user specifiable Backend. This way the unit test can inject the dependency it wants (namely a mock backend that tracks sub-chunk tables as they are acquired and released), we don't pollute the actual implementation classes with ""I'm fake"" flags and alternate code paths for fake objects, and we can make the unit test actually perform validity tests. The first cut at this should include a check for the problem described in DM-3522.",4 DM-3543,"Emergent Uncategorized Work","Epic to capture work that is not easy to categorize in other WBSs.",30 DM-3544,"Cleanup of initial astrometry improvements","The astrometry improvements are working, but some cleanup would be good to remove dependencies on A.net and to provide default reference catalog loaders.",20 DM-3545,"Improve aperture correction implemented in HSC","There is technical debt left over from the HSC-LSST merge of the aperture corrections. This epic will take care of the debt noted during the port.",15 DM-3546,"Move LDM-151 to Sphinx/Read the Docs","Move the LDM-151 (DM applications design document) to restructuredText (built with Sphinx) and published automatically via readthedocs.org. See discussion at http://community.lsst.org/t/requesting-comments-for-design-documentation-format-for-dm/132?u=jsick This is an experiment.",1 DM-3547,"SsiService not being destroyed","SsiService::~SsiService is not being called. ",1 DM-3548,"Learn about IPython / Jupyter internals","In order to evaluate the suitability of IPython as a framework for Level 3 work, and its ability to be integrated with the SUI Tools, the internal structure of IPython and its communication protocols need to be understood. Work on this story will include reading about IPython, experimenting with it, and reading the IPython code as needed.",14 DM-3549,"Further improve the TAN-SIP WCS fitter","lsst.meas.astrom.sip.makeCreateWcsWithSip, our C++ implementation of a TAN-SIP WCS fitter, fits for X and then Y. Thus it must be called several times (e.g. by FitTanSipWcsTask) in order to converge. Please make makeCreateWcsWithSip fit for X and Y simultaneously. This would simplify its use and speed up fitting. In addition, please investigate whether outlier rejection can be added to makeCreateWcsWithSip. As of DM-3492 outlier rejection is implemented in FitTanSipWcsTask, but it would make makeCreateWcsWithSip easier to use and speed up fitting if makeCreateWcsWithSip did the outlier rejection itself. On the other hand, the current solution may be sufficient. Once the above is implemented, please simplify FitTanSipWcsTask by removing the unneeded extra iterations used to work around these problems.",12 DM-3550,"Future measurement algorithm enhancements",NULL,40 DM-3551,"Attend SciPy 2015 tutorials","Attend the tutorials at SciPy 2015 (July 6-7, 2015) in order to get hands-on experience with current scientific data analysis tools in the Python environment. Planned attendance: * Introduction to NumPy * Building Python Data Applications with Blaze and Bokeh * Efficient Python for High-Performance Parallel Computing * Jupyter Advanced Topics Tutorial ",4 DM-3552,"Discuss the QA visualization needs","Meet with SQAURE lead to discuss the QA needs of visualization tools. The whole SUIT team of 8 people met with SQUARE lead Frossie for an all day discussion to outline the visualization needs that SQUARE team may need in support of the pipeline stack data processing verification. We agreed that the image visualization and XY 2D plot components should be separately accessible through JavaScript API and Python API. SQUARE team could build its own web portal using the JavaScript API and build its own analysis tool in iPython notebook using the Python API. The resulting work work was captured in other Epic and stories. ",18 DM-3553,"Access predefined catalogs via Data Access API","As a part of end-to-end exercise access predefined catalogs and their definitions from the new Data Access API. We have been doing it via JDBC calls to QServ and queries to http://lsst-web.ncsa.illinois.edu/schema/index.php?sVer=S12_sdss Even though we can not access QServ via Data Access API at the moment, it should be transparent to us in the future. As for data definitions, for now we can only access column names and types. In future, more information (like units and field descriptions) should be available.",10 DM-3554,"Binary FITS table catalog upload","We need to be able to upload catalogs in binary FITS table format. We'll do it by converting the first table in the provided FITS into an IPAC table. Our upload should be handling the conversion. Later, we should figure out how to handle multiple tables in one FITS.",4 DM-3555,"Ignore ""SELECT @@tx_isolation"" queries","Looks like one of the queries we registered in webserv is: cursor.execute('SELECT @@tx_isolation') and that is bound to confuse Qserv. Need to suppress it at mysql proxy level.",1 DM-3556,"Attend SciPy 2015 conference","Attend the SciPy 2015 conference, July 8-10, 2015, to learn about current trends in this community. Notably, the conference covers a variety of interactive data analysis tools, including Jupyter/IPython, interactive graphics packages such as VisPy and Bokeh, and astronomical data handling tools such as astropy.",6 DM-3557,"Document SciPy 2015 takeaways","Write up what was learned, and recommendations, from SciPy 2015.",4 DM-3558,"Experiment with Jupyter widget technology and Firefly Tools","Based on DM-2047 work to date, investigate the feasibility of using the Jupyter widget interface to wrap up Firefly tools.",8 DM-3559,"Implement improved Footprints","In S15/DM-1904 we began a redesign of the Footprint API. We need to follow through and convert that to code on disk.",60 DM-3560,"Complete HSC port: object characterization","Merge all functionality from HSC to LSST that is required so that the LSST stack can be used for all standard HSC processing functionality and the HSC fork is removed.",100 DM-3561,"Continued galaxy shear fitting experiments","Build on the test framework delivered in DM-1108 to arrive at conclusions regarding the number of pixels which must be included in galaxy fitting and the number of shapelet terms needed in the PSF.",60 DM-3562,"Refactor executor code",NULL,18 DM-3563,"Add unit tests to exercise new scheduler","Add tests (unit test and/or extend the integration test) to test the new scheduler.",10 DM-3564,"Integrate Qserv code with cancellation-friendly xrdssi",NULL,6 DM-3566,"Add support for type aliases",NULL,6 DM-3568,"Complete HSC port: framework","Merge all framework functionality (afw, middleware, etc) from HSC to LSST that is required so that the LSST stack can be used for all standard HSC processing functionality and the HSC fork can be removed.",60 DM-3569,"Research/reading for PAST prototype C++",NULL,6 DM-3571,"Make location of images more flexible","It'd be useful to be able to point imgserv to any location containing images (in particular for various random tests that we will be running over the next few years). Right now this requires changing imgserv code. An idea tossed around: pass the location via URI (optional parameter)",4 DM-3572,"Collect requirements for pipeline developer visualization tools","This story covers a series of conversations with Robert Lupton, Jim Bosch, Paul Price, K-T Lim, and others going into details about how to make the visualization environment (based on Firefly tools) more useful for developers.",4 DM-3573,"Document AP simulator","Write up how the AP simulator current works, i.e. start the processes doing the AP simulator, sending messages to the base DMCS, how messages are sent from archive to jobs on the worker cluster, etc.",10 DM-3574,"Replace qservAdmin.py use with CssAccess","WE have new CSS interface which unifies C++ and Python and it is time to replace qservAdmin.py with CssAcces in places like qserv-admin.py and data loader. ",8 DM-3578,"Research requirements for chromaticity",NULL,30 DM-3579,"Redesign CalibrateTask","This covers design work only; an accompanying epic (perhaps in 02C.03?) will handle implementation.",10 DM-3580,"Refactor sub-task interfaces",NULL,50 DM-3581,"Audit, update & integrate top level tasks","As part of the HSC merge, we've pulled in a number of high-level pipeline tasks, which may not be consistent with existing LSST tasks. Audit the overall pipeline flow, ensuring consistency. To include: * Rewrite top-level pipeline tasks and Butler datasets. * Remove ProcessCoaddTask. * Ensure consistency between MPI drivers and individual CmdLineTasks.",25 DM-3582,"Investigate options for physically motivated PSF models","Rather than developing PSF models based on the optics and the atmosphere, this epic is focused on developing requirements and establishing what resources & expertise are available in other groups (e.g. DESC) which we can make use of. If necessary, it will flow down to another epic in which we actually produce working code.",30 DM-3583,"Develop improved galaxy model flux measurements","Hard thinking, cleaning up & optimizing the existing galaxy model flux measurement code.",50 DM-3585,"DRP W16 emergent work: object characterization","Catch all epic for emergent work in 02C.04.06 during W16.",20 DM-3586,"DRP W16 emergent work: framework","Catch all epic for emergent work in 02C.04.01 during W16.",30 DM-3587,"Firefly infrastructure improvement to support new functions (W16)","This epic will capture the necessary changes of Firefly infrastructure to support new functions needed. It does not include the changes caused by the the conversion from GWT infrastructure to pure JavaScript based system using React and FLUX platform. ",40 DM-3588,"Refactor the Firefly Java code (W16)","This epic will capture all the refactoring work related to Java code. We are converting portions of GWT code to pure JavaScript React based code. We will only refactor the Java code if it is not going to be converted. ",30 DM-3590,"Catalog transformation should pass through all non-measurement fields","We need to include fields added by other tasks (e.g. the deblender) or the source minimal schema (e.g. parent) in transformed catalogs. I believe it should be safe to assume any such fields can simply be copied (i.e. they do not require any actual transformation - they're mostly flags), but we do need to make the list of fields to be copied by this mechanism dynamic.",3 DM-3591,"Add support for registry-free repository","Existing butler unit tests should run without an sqlite database registry.",12 DM-3592,"Refactor Backend to improve visibility","Backend is a class that connects to mysql and can cause the worker to terminate. It is buried in the Foreman and should be renamed and have the class defined in a header file so that it is more visible. The class should also be available to use for any other in memory tables. ",10 DM-3593,"Firefly support for pipeline visualization needs (W16)","Data products pipeline needs visualization capabilities for display. Firefly needs to have new capabilities to support it. ",40 DM-3595,"New functions for XY 2D plot (W16)","Firefly should support histogram plot",50 DM-3596,"new algorithm and functions for 2D XY plot (F17)",NULL,30 DM-3597,"use devtoolset-3 for Jenkins CI builds on EL6",NULL,2 DM-3599,"Firefly support for Camera team visualization needs (W16)","Camera team plans to use Firefly visualization capabilities in camera test. SUI/T team will continue to support this effort. ",30 DM-3600,"Workspace preliminary functional design (W16)","Produce a preliminary functional design of user workspace. ",18 DM-3601,"Authentication and authorization system API requirement","Produce a requirement document on the authentication and authorization API needed for the full SUIT. Work with other DM subsystems (DB, SQUARE, NCSA) closely to achieve this. ",10 DM-3602,"Prepare for system setup in NCSA for LSST web UI (X16)","We want to start preparing for the SUI system setup in Fall 2016 at NCSA to be able to do the following: Set up a web UI for DM or even general public to access, so we can have a constant end-to-end system. It will be very useful for DM testing. SUIT will use it for simple test of accessing the DAX APIs. Pipeline team can use it to see the processed data, search source table etc. The goal is to be able to access SDSS strip 82 data through SUIT web portal",10 DM-3603,"Expose more Firefly visualization functions through JavaScript API (W16)","Expose more Firefly visualization functions through JavaScript API to users so they can have more control in building their own web page. ",10 DM-3604,"Expose more Firefly visualization functions through Python API (F16)","Expose more Firefly visualization functions through Python API as needed.",22 DM-3605,"Data access (DAX) API design support (W16)","Continue to work with SLAC in data access (DAX) API design and test. Participating the discussion and exercising the API as needed to help test and expose potential issues. ",10 DM-3606,"Submit change request for LSE-68, mid-phase-3 update","Collect changes and submit a change request to LSE-68 for a mid-phase-3 update to the document, covering clarifications on guider data and image identification, among others. Includes preliminary work to prepare for change request.",4 DM-3607,"Prepare document for CCB review of LCR-357","LCR-357 was an outline of work to be done, based on discussions with the Camera DAQ staff. This task is to generate an actual proposed document change for the CCB.",6 DM-3608,"provide detailed information needed to DAX meta API","SUIT needs certain specific information through DAX meta service when searching for meta data. For Example, what kind of table it is, does it have spatial index to search by position, which set of (ra, dec) columns is the primary one, etc?",1 DM-3609,"The Alert subscription system requirement gathering (F16)","Solidify the requirement for the alert subscription system. ",8 DM-3610,"CCB review and posting of final updated document","Carry out the CCB review, respond to questions, support final implementation of updated document.",2 DM-3611,"Prepare for Winter 2016 work on LSE-68","Use a session at the LSST 2015 all-hands meeting to prepare for LSE-68 work in the Winter 2016 cycle.",3 DM-3612,"option to plot error bars on XY plot","When there are error or uncertainty for a data point, there should be option to plot the error bars in the XY plot. ",6 DM-3613,"Time series plot","Time series plot",10 DM-3614,"capability to reverse the axis","Provide option to reverse the axis for XY plot, one example is the magnitude value convention. ",10 DM-3615,"expose region overlay on image function through JavaScript API","expose region overlay on image function through JavaScript API",5 DM-3616,"Expose image XY readout at cursor point function in JavaScript API","Expose image XY readout at cursor point function in JavaScript API",2 DM-3617,"Review risk register status",NULL,3 DM-3618,"Fix bug related to restarting xrootd in wmgr","Changes from DM-2930 are failing integration tests because wmgr is restarting xrootd and now we need to also restart mysqld if xrootd pid changes.",1 DM-3619,"Move env variables related to xrootd/czar and unlimit in etc/sysconfig/qserv",NULL,1 DM-3620,"OCS-DM-CCS-DAQ workshop","Prepare for and attend the OCS-DAQ-CCS-DM workshop in November 2014. This is primarily intended to review the status of LSE-70 (the definition of the OCS interface to the subsystems) and therefore prepare for updates to LSE-72 in time.",6 DM-3621,"OCS-DM-CCS-DAQ workshop II","Prepare for and attend the OCS-DAQ-CCS-DM workshop in February 2015. This is primarily intended to continue to make progress on LSE-70 (the definition of the OCS interface to the subsystems) and therefore prepare for updates to LSE-72 in time.",6 DM-3622,"Overhaul deblender","Re-write the existing deblender code to use the refactored Footprints (DM-3559). Take the opportunity to resolve known issues and add new functionality from SDSS and elsewhere to make the deblender equal to the state of the art.",60 DM-3623,"LDM-144 costing model update",NULL,6 DM-3624,"Investigate disk-only (no tape) data releases","The question that is being raised is how much it would cost to keep more than the last two Data Releases on readily-accessible storage (i.e. spinning disk). This will require changing several numbers and formulas in LDM-141, the storage sizing model, observing the results as they propagate through LDM-144, the cost model, and checking to make sure that everything makes sense and nothing has been overlooked. It looks like an answer to this question will be needed somewhere between a month and three months from now.",2 DM-3625,"Continue to refine the SUIT requirements (X16)","We did lots of work in FY15 to redefine the SUIT requirements. The picture is getting clear as we talk to others in DM system and science community. We would like to have a better definition and provide a document for this. ",10 DM-3626,"a catch all epic for unexpected bug fixes (W16)","This is an epic for unexpected bugs found and need to be fixed in this cycle. ",10 DM-3627,"implement jenkins support for running builds in docker containers","DM-3359 demonstrated the feasibility of running jenkins builds in one-off docker containers but did not cover investigation of user creation of containers nor did it include a puppetized deployment. Per build containers are desirable for a number of reasons but the largest motivation is allowing DM developers to define their own jobs without disrupting the ""sanctity"" of the lsstsw build slaves.",7 DM-3628,"HSC port: verification","Compare the LSST and (old) HSC stacks for compatibility; identify and resolve any differences. Provide basic integration tests to SQuaRE to ensure the reliable and consistent operation of the LSST stack for processing HSC data. Cooperate with the SQuaRE team to develop more elaborate QA testing. This work is timeboxed to consume the available effort in W16, and will be continued in the subsequent cycle.",50 DM-3630,"Change root to config in config override files","Implement RFC-62 by using {{config}} rather than {{root}} in config override files for the root of the config. Note that I propose not modifying astrometry_net_data configs because those are numerous and hidden. They have their own special loader in LoadAstrometryNetObjectsTask._readIndexFiles which could easily be updated later. if desired. An obvious time to make such a transition would be when overhauling the way this data is unpersisted.",2 DM-3631,"Management for Aug 23-29","Hiring -- make offer to candidate, and also review candidates and strategy. Review proposal for FY 2015 spend Management meetings. Openstack meeting.",3 DM-3632,"Study JavaScript, read up on React ","Spend about one hour per day to read up on JavaScript and React framework. Get ready to work on some of the JavaScript programming in Firefly",10 DM-3633,"Add readMatches back to meas_astrom","Recent changes to meas_astrom accidentally removed a function readMatches (copied below). Please restore it, preferably in its own module (though if someday we have more small python functions we may want a utils.py module). Also please include a unit test. {code} def readMatches(butler, dataId, sourcesName='icSrc', matchesName='icMatch', config=MeasAstromConfig(), sourcesFlags=afwTable.SOURCE_IO_NO_FOOTPRINTS): """"""Read matches, sources and catalogue; combine. @param butler Data butler @param dataId Data identifier for butler @param sourcesName Name for sources from butler @param matchesName Name for matches from butler @param sourcesFlags Flags to pass for source retrieval @returns Matches """""" sources = butler.get(sourcesName, dataId, flags=sourcesFlags) packedMatches = butler.get(matchesName, dataId) astrom = Astrometry(config) return astrom.joinMatchListWithCatalog(packedMatches, sources) {code} ",4 DM-3634,"Preliminary design of Firefly core using React and FLUX framework","Propose a preliminary design for Firefly core using React and FLUX framework. ",12 DM-3635,"configDictField.py has code that relies on an undefined variable","While taking a linter pass on {{pex_config}} I found that {{ConfigDict.\_\_setitem\_\_}} in {{configDictField.py}} has some code that uses an undefined variable {{value}}. See the else clause in: {code} if oldValue is None: if x == dtype: self._dict[k] = dtype(__name=name, __at=at, __label=label) else: self._dict[k] = dtype(__name=name, __at=at, __label=label, **x._storage) if setHistory: self.history.append((""Added item at key %s""%k, at, label)) else: if value == dtype: value = dtype() oldValue.update(__at=at, __label=label, **value._storage) if setHistory: self.history.append((""Modified item at key %s""%k, at, label)) {code}",2 DM-3636,"continue L1 refined specifications ","Re-synchronize with prior work after vacation. Write up page both engineering and facility database ingest and use, incorporate suggestion and comments from KT and GDF. Clean up (better name entities, and move for better narrative flow) pages about the general thing that is called ""level 1"", -- the context diagram and supporting prose for further descriptions, Review internally with Jason, Steve and Margaret. Create a revised diagram that would change LDM-230 specifications. (all non crosstalk corrected data flow into a disk buffer, and and an new archive tasks empties it) to make specifications consigned with the refined specifications on the page above. Yet to write the prose.... These pages are in my personal pages in the LSST confluence area. ",4 DM-3637,"preliminary detailed content required for Authorization and Authentication system for SUIT","Provide the first draft of detailed content required for authorization and authentication system from SUIT point of view to NCSA. ",6 DM-3638,"RangeField mis-handles max < min","RangeField contains the following bit of code to handle the case that max < min: {code} if min is not None and max is not None and min > max: swap(min, max) {code} This is broken because there is no swap function and if there was it could not work in-place like this. However, rather than replace this with the standard {{min, max = max, min}} I suggest we raise an exception. If max < min then this probably indicates some kind of error or sloppiness that should not be silently ignored. If we insist on swapping the values then at least we should print a warning. The fact that this bug has never been reported strongly suggests that we never do set min > max and thus that an exception will be fine.",1 DM-3639,"OCS-CCS-DAQ-DM teleconference, April 2015","Prepare for and attend a half-day teleconference on OCS issues.",2 DM-3640,"OCS-CCS-DAQ-DM workshop III, May 2015","Prepare for and attend an OCS-subsystems workshop at SLAC May 6-8, 2015.",6 DM-3641,"Firefly server side extensions using DM stack (F16)","Design and implement a control system to extend Firefly server side capabilities using task in DM stack. This will make it easier to use DM stack for customized data processing. ",40 DM-3642,"Support OCS revision of LSE-70, LSE-209","Support the OCS efforts to update LSE-70 and create a new associated document, LSE-209. Getting current versions of these under change control will allow us to complete a round of work on LSE-72.",20 DM-3643,"install DM stack, get familiar with the current DM task concept ","install DM stack, get familiar with the current DM task concept. This is for getting ready to use task with Firefly server side extension capability. Here is the link to the tutorial. https://confluence.lsstcorp.org/display/DM/Getting+started+with+stack+development",6 DM-3644,"Support the design of Firefly core system using React and FLUX","Working with Loi on the design of Firefly core system based React and FLUX frameowrk",4 DM-3645,"Support the design of Firefly core system using React and FLUX ","Working with Loi on the design of Firefly core system based React and FLUX frameowork ",4 DM-3646,"LSE-72: OCS-CCS-DAQ-DM workshop, July 2015","Work associated with Workshop IV in the series, held at NCSA July 8-10, 2015.",2 DM-3647,"Preliminary SUIT design ","Working with the suit-wg to produce a preliminary design of SUIT.",54 DM-3648,"SUIT design document outline","SUI/T design document outline. ",2 DM-3650,"on-going support to Camera team in UIUC","Attend UIUC weekly meeting and give support as needed. ",2 DM-3651,"MakeDiscreteSkyMapRunner.__call__ mis-handled returning results","{{MakeDiscreteSkyMapRunner.\_\_call\_\_}} will fail if {{self.doReturnResults}} is {{True}} due to trying to reference undefined variables. This is at least approximately a copy of a problem that was fixed in pipe_base {{TaskRunner}}. {{MakeDiscreteSkyMapRunner.\_\_call\_\_}} should be fixed in a similar way, and (like {{TaskRunner}}) changed to return a pipe_base {{Struct}}. ",2 DM-3652,"SUIT design document outline","Work with Gregory on the SUIT design document outline 1. Requirements flow down, making sure that we design the system satisfying the current requirements. 2. Use cases collection. at least one typical use case in each major science theme 3. Levels of different users ** novice: treat the web portal as a archive to get some information, don't know much about LSST ** novice expert: has some ideas of what special functions they would like, has some knowledge of LSST data ** domain expert: knows LSST data very well and want some special functions ready to use ** savvy expert: knows LSST data very well and like to use API to their own programming 4. functions for all different levels of users 5 system design ** system diagram ** details of the different parts *** Firefly server *** Firefly client *** Firefly server extension *** Firefly JavaScript API *** Firefly Python API *** Firefly Python API, Jupyter notebook, and other Python applications *** workspace and level3 data *** SUI web portal sketch, workflow ** dependency on other capabilities of other institutes 6. development and test plan, timeline 7. deployment plan ",1 DM-3653,"SUIT design document outline","work with John Rector on SUIT design document outline",1 DM-3654,"Summarize current LSE-75 status as intro for new T&S personnel","With the arrival of new Telescope & Site personnel, especially the Telescope Scientist, [~sthomas], prepare a summary of the current state of LSE-75 and its open issues.",4 DM-3656,"Data loader doesn't work for match tables","qserv-data-loader.py fails to load match tables: - it does not invoke the correct partitioner executable for them - not all CSS parameters required for match tables are passed down to the CSS update code",1 DM-3657,"Create change request for LSE-75","Create a change request for LSE-75, the TCS - to - DM ICD.",2 DM-3658,"Discussions on LSE-75 with Telescope & Site personnel","Pursue interactions with Telescope and Site personnel regarding LSE-75, and in particular the issues surrounding calibration data products for the wavefront and guider data analysis pipelines. Covers work through the end of August 2015.",3 DM-3659,"Initial discussions with Patrick Ingraham","This story is a catch-all for preliminary conversations about LSE-140 with the new Calibration Instrumentation Scientist, Patrick Ingraham.",1 DM-3660,"Review pending work, clean up related JIRA DM- and LIT- issues",NULL,2 DM-3661,"Configure VMs to provide additional slots for task switching","We have to set up a new set of slots that will be used to execute the overlapping thread of execution for alert production. Because of limited resources at this time, this means reconfiguring the worker nodes to provide additional slots, and to split the worker nodes into two sets. ",3 DM-3662,"Add support for clang and OS X to qserv scons",NULL,2 DM-3663,"Basis for HSC integration test","We need to assemble the basis for an integration test using HSC data, to protect HSC processing and obs_subaru from upstream changes. This includes the assembly of the required data, a basic mechanism to test the mechanics of data release production, sufficient for the SQuaRE team to take it and incorporate it into the Jenkins system. Addition of any scientific validation is deferred for now.",4 DM-3664,"rename parameter vector methods in afw.geom.ellipses","[~nlust] notes that the {{writeParameters}} and {{readParameters}} methods on the ellipse classes are confusingly named, especially when compared to similar methods on {{meas.modelfit.Model}}.",2 DM-3665,"improve test coverage of CModel failure modes","The CModel has a large number of failure modes, largely dealing with different kinds of problems in the inputs, and a correspondingly large number of flags. It also has some fairly complex logic determining which flags can be set simultaneously. All of these combinations need to be tested. DM-1574 may be useful in capturing these conditions from runs on real data.",6 DM-3666,"Revisit Footprint design",NULL,4 DM-3667,"PSFEX does not build if PLplot is installed","During the configure phase PSFEX checks for the presence of PLplot. If PLplot is found then the build fails (at least on a Mac using homebrew): {code} /bin/sh ../libtool --tag=CC --mode=link clang -g -O2 -I/Users/timj/work/lsstsw/src/psfex/lapack_functions/include -o psfex check.o context.o cplot.o diagnostic.o fft.o field.o field_utils.o fitswcs.o homo.o main.o makeit.o makeit2.o misc.o pca.o prefs.o psf.o sample.o sample_utils.o vignet.o wcs_utils.o xml.o ./fits/libfits.a ./levmar/liblevmar.a ./wcs/libwcs_c.a -L/Users/timj/work/lsstsw/stack/DarwinX86/fftw/3.3.3-1-g8fdba61+da39a3ee5e/lib -lfftw3f -lm -L/Users/timj/work/lsstsw/src/psfex/lapack_functions/lib -llapackstub -lf2c -lm -lplplotd libtool: link: clang -g -O2 -I/Users/timj/work/lsstsw/src/psfex/lapack_functions/include -o psfex check.o context.o cplot.o diagnostic.o fft.o field.o field_utils.o fitswcs.o homo.o main.o makeit.o makeit2.o misc.o pca.o prefs.o psf.o sample.o sample_utils.o vignet.o wcs_utils.o xml.o ./fits/libfits.a ./levmar/liblevmar.a ./wcs/libwcs_c.a -L/Users/timj/work/lsstsw/stack/DarwinX86/fftw/3.3.3-1-g8fdba61+da39a3ee5e/lib /Users/timj/work/lsstsw/stack/DarwinX86/fftw/3.3.3-1-g8fdba61+da39a3ee5e/lib/libfftw3f.dylib -L/Users/timj/work/lsstsw/src/psfex/lapack_functions/lib -llapackstub -lf2c -lm -lplplotd Undefined symbols for architecture x86_64: ""_plwid"", referenced from: _cplot_drawloccoordgrid in cplot.o _cplot_fwhm in cplot.o _cplot_ellipticity in cplot.o _cplot_moffatresi in cplot.o _cplot_asymresi in cplot.o _cplot_counts in cplot.o _cplot_countfrac in cplot.o ... ld: symbol(s) not found for architecture x86_64 {code} This particular error is caused by PSFEX using a deprecated PLplot API ({{plwid}}) that is not enabled by default and whose name is not translated to {{c_plwid}}. This PLplot change occurred in version 5.9.10 released in 2013. I assume upstream PSFEX has a fix for this. Given that LSST does not need the PLplot functionality I think the simplest fix may well be to disable the test for PLplot in our version. It seems likely that there will be a reasonable number of systems ""in the wild"" who will have PLplot installed so I'm inclined to think that this should be a blocker for the v11.0 release. If we are lucky people will have all upgraded their PLplot installs to v5.11.0 because in that version PLplot change the name of the library from {{libplplotd}} to {{libplplot}} and PSFEX has hard-wired the former rather than using pkg-config. This results in configure not finding PLplot. I don't think this eventuality is likely though. ",1 DM-3670,"obs_test needs to override map_camera and std_camera","The Butler can't get a camera unless the map_camera and std_camera are defined correctly. In most cases the camera can be built by the map_camera method. In the case of obs_test, the camera is built in the constructor of the Mapper, so std_camera should just return the camera attribute.",1 DM-3672,"W16 Webserv Unit Tests","Implement unit testing across the webserv components. Use either a SQLite backend, or mock objects and mock results to emulate database interaction.",10 DM-3673,"Gather candidates for Verification Datasets and identify collaborators"," Identify candidate Verification sets (in the first instance, datasets with extant processed data used in one-off reductions with lsst_apps to allow a preliminary assessment of current quality from a science QA perspective). Identify people within the project with effort, expertise and interest available to contribute to this effort. ",10 DM-3674,"Present Verification approach at AHM"," Present talk at AHM to seed Verification effort and seek feedback. ",10 DM-3675,"Resourcing Verification runs"," Identify required resources for Verification runs and communicate them to NCSA. ",2 DM-3677,"HSC backport: Cleanup interpolation tasks and implement useFallbackValueAtEdge","This is a port of the changesets from: [HSC-756|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-756] ",4 DM-3678,"HSC backport: Standalone updates to star object selection","This involves pulling over the following standalone (i.e. non-ticket) HSC commits: [Updated star selection algorithm.|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/071fcadc016908a10583c746f0a8e79df2a45ead] [Appropriate config parameter for a unit test of testPsfDetermination.py.|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/e73c5e447ac0b8a71926d3e78fec30aad4beee91] [Remove HSC specific codes.|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/15bb812578531766199e9a1ee41cc707fb3d9873] (Note, the above reverts some unwanted camera-specific clauses added in the first commit. May just squash them to only add the desired features) [ObjectSizeStarSelector: push non-fatal errors to DEBUG level|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/44f75bc60b41c5f77b323a8d9981048ef7e5f3c4] [We don't use focal plane coordinates anywhere, and detector may be None|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/4413db4610e4793727e591f395f5ad8cd0cb6030] [Fixed axis labels|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/67efacaccf8346fdfa1b450617aebabddb2b7ec0] [Improved PSF debugging plots|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/b1bc91ed1538607eb90e070881a82498fd551909] [Worked on star selector|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/6b36f4d757187d30142a7e026754a07ffeb8dea2]",1 DM-3679,"Allow building/publishing components off branches other than master","Support of xrootd within the stack is currently complicated by the fact that qserv depends on features that are not available on upstream master (only available on an upstream non-master branch). Since we can currently only publish packages from master, this means that our lsst fork of xrootd cannot be a ""pure"" fork -- we end up merging/rebasing from an upstream branch, then force-pushing the downstream master. Upstream and downstream xrootd repos thus have completely different branch topologies, labels, etc., and history of master in the lsst fork is being continually rewritten to carry local patches forward. The processes of both adopting upstream changes into the lsst fork and the pushing lsst changes back upstream are cumbersome, confusing, and labor intensive. It is proposed that we extend our tools to allow publishing components from branches other than master. This would allow us to have xrootd for example be a ""pure"" fork of upstream -- we could then create our own branch based off any upstream branch, carry our downstream patches there, and release off of that. This functionality could be used similarly for any of our current ""t&p"" components where it would be convenient to track the upstream repo directly and/or carry changes in git instead of in an agglomerated patch file (e.g. when we might want to update frequently and/or contribute general purpose changes back upstream regularly with pr's, etc.)",2 DM-3681,"Jira PMCS EVM integration","This epic captures management support requests and non-SPed activities in FY15 covered by WBS 02C0102 that were not included in other Epics. ",100 DM-3682,"LDM-240 Long range planning",NULL,100 DM-3683,"Jira Data Management long range planning project",NULL,100 DM-3684,"Release engineering Part Two","This epic covers testing and co-ordination work associated with making engineering and official releases, and code to support them. [FE at 70%, JH at 20%, JS at 10%]",40 DM-3685,"Evaluate GitLFS","We want to try out GitLFS and evaluate its suitability as a solution to storing binary data in repos for CI. We want to know how it compares in usability terms to git-annex, git-fat etc. ",17 DM-3686,"Fix PATH and compiler version detection in qserv scons","In recently merged DM-3662 compiler version testing was done using OS tools with regular $PATH. This is inconsistent with other scons tools which reset PATH when executing actions. We want to do two things: - propagate PATH to the command execution - Use scons tools to run ""$CXX --version"" instead of OS tools to keep things consistent",1 DM-3687,"Revisit KPIs for Image Access","Need to come up with KPIs for Image Query Access",4 DM-3688,"lsst_dm_stack_demo failure","Viz: {code} $ ./bin/demo.sh [...] $ bin/compare detected-sources.txt.expected detected-sources.txt Failed (max difference 0.439326 over tolerance 0.004000) in column base_GaussianFlux_flux. Failed (max difference 0.439326 over tolerance 0.004000) in column base_GaussianFlux_fluxSigma. Failed (max difference 0.244707 over tolerance 0.004000) in column base_PsfFlux_flux. Failed (max difference 0.244707 over tolerance 0.004000) in column base_PsfFlux_fluxSigma. {code} This is on OS X with {{lsst_apps}} {{w_2015_36}}. ",2 DM-3690,"Forward community.lsst.org (Discourse) notifications to existing mailman lists","Setup a system to forward new post notifications from http://community.lsst.org categories to their appropriate legacy Mailman email list counterparts. ||Discourse Source Category||Mailmain Forward List|| |DM Team||dm-staff| |Announcements||dm-announce| |DM Notifications||dm-devel| Once this is implemented I will send a notice that all communications and replies should occur on discourse (these mailing lists should be read-only). I will also send a notice that dm-users is deprecated. ",7 DM-3691,"CalibrateTask has outdated, incorrect code for handling aperture corrections","The CFHT-specific CalibrateTask tries to apply aperture correction once just after measuring it (which is too early) and again later, at the right time. The error probably has no effect on the final results, but it is confusing and needlessly divergent from the standard CalibrateTask. The required changes are small. I plan to test by running [~boutigny]'s CFHT demo.",1 DM-3692,"HSC backport: Allow for some fraction of PSF Candidates to be reserved from the fitting","This is a port of the changesets from [HSC-966|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-966]. It provides the ability to reserve some fraction of PSF candidates from the PSF fitting in order to check for overfitting and do cross validation.",1 DM-3693,"HSC backport: allow photometric and astrometric calibrations to be required","This is a port of the standalone changesets: [calibrate: make astrometry failures non-fatal|https://github.com/HyperSuprime-Cam/pipe_tasks/commit/e9db5c0dcdca20e8f7ba71f24f8b797e71699352] [fixup! calibrate: make astrometry failures non-fatal|https://github.com/HyperSuprime-Cam/pipe_tasks/commit/c2d89396923f9d589822c043ed8753647e70f3f6] (the above is a fixup, so will likely be squashed) [make failure to match sources non-fatal|https://github.com/HyperSuprime-Cam/pipe_tasks/commit/cf5724b852937cfcef1b71b7a372552011fda670] [calibrate: restore original Wcs after initial astrometry solution|https://github.com/HyperSuprime-Cam/pipe_tasks/commit/ab6cb9e206d0456dc764c5ef78ac80ece937c610] [move CalibrateTask from ProcessImageTask into ProcessCcdTask|https://github.com/HyperSuprime-Cam/pipe_tasks/commit/08a8ec029dd52ac55e47b707a6905df061a40506] [processCoadd: set detection to use the declared variances|https://github.com/HyperSuprime-Cam/pipe_tasks/commit/9e8563fd8d630dad967786387b1f27b6bc7ee039] [adapt to removal of CalibrateTask from ProcessImageTask in pipe_tasks|https://github.com/HyperSuprime-Cam/obs_subaru/commit/52733a7ab1731a15cbb93151851f57cec276f928] and HSC tickets: [HSC-1085: background not saved in processCcd|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1085] and [HSC-1086: psf - catalog scatter is very large in some coadds|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1086]",2 DM-3694,"Decrease buildbot frequency","Buildbot frequency is now down to two builds, one at 19:42 machine time (NCSA) and one at 1:42. This is to stop people needing buildbot runs to eups publish to have to wait before a CI build, since they are now done on[ https://ci.lsst.codes ]/ Jenkins. ",1 DM-3695,"Add unit tests for secondary index","qproc::testIndexMap.cc is very sketchy and doesn't perform any test for now (i.e. no BOOST_CHECK). It should be improved to really cover code related to secondary index. A mock secondary index is required here, i.e. qproc::FakeBacken should be strengthened.",8 DM-3697,"Install squid proxy on cc-in2p3 build node","Can we add this one to current sprint? It is required to access docker hub on in2p3 cluster. I also need to automate/document it, test it on build nodes, and be reviewed by in2p3 sysadmins. Cheers",4 DM-3698,"Replace --trace with --loglevel in pipe_base ArgumentParser","Replace the --trace argument with an enhanced version of --loglevel that supports named values and numeric log levels (which are the negative of trace levels). This simplifies the interface for users and potentially reduces the log level/trace level confusion, though that won't fully happen until we finish replacing use of pex_logging Trace and Debug with Log. This work was already done as part of DM-3532; it just needs to be copied with minor changes (since there are no named trace levels in pex_logging).",1 DM-3700,"Literature search for DCR -- Sullivan","Go through the literature to find relevant seminal papers on DCR. The outcome will be a bibliography and executive summary. This should be posted on Discourse.",15 DM-3701,"Setup and conduct a conversation about DCR in the project","Advertise and conduct a broadly advertised videocon on DCR in the context of diffim. The result of this should be minutes. Ideally we would come out of this meeting with a list of possible techniques for dealing with DCR (preferentially sorted by priority).",4 DM-3702,"Assemble the report on DCR","Everything learned through the literature search and project wide meeting should be synthesized into a single readable report that details the expected effects of DCR on difference imaging as well as possible mitigation techniques. This may involve some preliminary analysis work to measure effectiveness of various techniques. Note that I expect this to be two weeks of work for two people, thus the 40 story points. I don't know how to assign a story to two people.",40 DM-3703,"Port suspect pixel flags to meas_base","Pull HSC pixelFlags for suspect and suspect center over from {{meas_algorithms}} to {{meas_base}}. Additionally there are a few places in {{meas_base}} (& possibly in {{_algorithms}} as well) which set flags that have comments such as ""Set suspect flag if it was available"". Each of these places should be updated to use the ported bit. The relevant commit can be found at https://github.com/HyperSuprime-Cam/meas_algorithms/commit/21be65187c30302abb430d59fc5f67730ca7e0a1 and is discussed at https://dev.lsstcorp.org/trac/ticket/2838 It may also be necessary to add or update a unit test to make sure the flag is set appropriately.",4 DM-3704,"Refactor ImageDifferenceTask: Split into two tasks","This issue is to modernize the diffim task. Once it is running it should be refactored to use the new measurement tasks and reference loading wherever possible. Also remove any one off code used in past reports. Measurement should be split into its own task. The resulting task should be general, but not so general as to make it hard to run. This issue is being split into three issues: * DM-3704: split monolithic task into: ** ImageDifferenceTask - creates image difference ** ProcessDiffim - perform detection and measurement on image differences (and calexps) * DM-5294 Refactor/Clean up new ImageDifferenceTask * DM-5295 Refactor/Clean up new processDiffimTask",10 DM-3706,"fix EventLog references in ctrl_orca","There are a couple references to EventLog in ctrl_orca, which is an object that no longer exists.",4 DM-3707,"qserv scons - do not copy files to variant_dir","Some people are not happy with our current scons setup which copies source files from source directories to variant_dir, it makes it harder to trace errors using tools like eclipse or debug code. Would be nice to get rid of the extra copy, but we still want to have separate build directory (variant_dir). It should be simple enough, I think, but will need some testing of course.",2 DM-3749,"Scons build of lapack_functions in PSFex fails if SCONSFLAGS are set","The scons build system is unaware of extra flags which may be set in SCONSFLAGS environment variable, which are used from scons utils. This will cause the build to fail. The package needs to behave properly and build in the presence of these flags",2 DM-3750,"Prototype DRP sequence using DECam data","Learn the LSST stack and prototype a sequence for Data Release Production using DECam/DES data as inputs. Assignees: Hsin-Fang Chiang, Robert Gruendl Duration: September 2015 - February 2016",100 DM-3751,"Refine design specification and requirements analysis of Level 1 and Level 2 systems","Review of existing design documentation and gathering of requirements of all Level 1 & 2 (and 3) systems. Specify both a functional and physical breakdown of the systems for long-term planning and for building the production infrastructure needed at NCSA. Assignees: Don Petravick, Jason Alt, Paul Wefel, Steve Pietrowicz, Margaret Gelman Duration: September - December 2015",75 DM-3753,"Margaret's mgmt. activities in August","Local coordination meetings DMLT meetings LSST2015 workshop and conference Hiring/interviewing Budget review and FY16 prep Invoice re-breakouts July TPR etc.",28 DM-3755,"Revisit shared scans design",NULL,10 DM-3756,"Implement feature sets requested by SUI and DRP processing in Process Execution Framework","Extend the Process Execution Framework with feature sets requested by SUI and DRP processing. Assignees: Matias Carrasco Kind",38 DM-3757,"Operations planning for Archive and US DAC in TOWG/TPWG","Develop use cases and plan for operations in the Technical Operations Working Group and Technical Proposal Working Group. Assignees: Don Petravick, Margaret Gelman, Chris Pond, Robert Gruendl Duration: September 2015 - January 2015",58 DM-3758,"Expose table metadata via metaserv","SUI team would find it useful to get counts of columns for a table, ideally, all counts of columns for all tables in a given database in one request. They'd also find it useful if we could send column description.",6 DM-3759,"Analyze Qserv KPIs","Improve script for analyzing KPIs, measure and document KPIs.",24 DM-3760,"Support operation of development infrastructure (lsst-dev and other)","Administrative support to operate the LSST development cluster and LSST's use of the NCSA OpenStack nebula. Includes planned maintenance activities and LOE/emergent work. Expect ~ 1FTE day per week. Assignees: Bill Glick, Matt Elliot, Bruce Mather, Paul Wefel Duration: September 2015 - February 2016",49 DM-3761,"Sizing model storage costing update","The question that is being raised is how much it would cost to keep more than the last two Data Releases on readily-accessible storage (i.e. spinning disk). This will require changing several numbers and formulas in LDM-141, the storage sizing model, observing the results as they propagate through LDM-144, the cost model, and checking to make sure that everything makes sense and nothing has been overlooked. Assignees: Jason Alt Duration: September 2015",6 DM-3762,"Refine file system policies and services","Refine data management policies and services (e.g., data retention, backup policies). Ideally we would have a draft of this by the November DMLT meeting. Assignees: Jason Alt, James Parsons Duration: October - November 2015",13 DM-3764,"Czar dies when parser throws exception","Running a query that mistakenly uses scisql_s2PtInBox instead of qserv_s2PtInBox {code}select objectId, coord_ra, coord_dec from smm_bremerton.deepCoadd_forced_src where scisql_s2PtInBox(coord_ra, coord_dec, 320.05, 0.457, 320.06, 0.46){code} kills czar, the error message in czar log file is: {code}ERROR ccontrol.UserQueryFactory (build/ccontrol/UserQueryFactory.cc:117) - Invalid query: ParseException:Parse error(ANTLR):unexpected token: scisql_s2PtInBox:{code} We need to change the code so that a random query does not kill czar. This story involves fixing czar so that it does not die when parser chokes on the syntax. ",6 DM-3765,"For registry-free butler, look up information in related data type.","Butler reading information (in particular the observation time and length) out of an input dataset's file representation in order to provide rendezvous with calibration data in another repository (that does have a registry). Today, that read is handled by genInputRegistry.py so that the butler doesn't need to look into the dataset itself. If there's no registry, such a read will be necessary. The ultimate test is to run processCcd.py (a CmdLineTask that runs Instrument Signature Removal) on a repository that contains raw frames that require more than one set of calibrations and have it pick up the correct ones.  That would have to be condensed down into a unit test.",12 DM-3766,"Add Butler access to calibration data in obs_decam","Goals: - Have something to ingest calibration data and create a calibration registry (calibRegistry.sqlite3) - Add mapping class calibration into DecamMapper and make butler able to get bias/flat/fringe. - Include calib in testdata_decam and add to the unit test retrieving calib data. Summary: - A new task {{ingestCalibs.py}} is added to {{pipe_tasks}} for parsing through calibration data and creating a calibration registry. The DECam customized configuration is added in {{DecamCalibsParseTask}} in {{obs_decam}}. {{pipe_tasks/ingest.py}} is modified slightly for more general use while its original behaviors are kept by default. - The calibration data are DECam Community-Pipeline products, including nightly-MasterCal bias/flat downloadable from NOAO Science Archive, and fringe of the latest version 56876 from http://www.ctio.noao.edu/noao/content/decam-calibration-files or also on /lsst8/decam/cal/DECamMasterCal_56876/ Note that files from the two sources have different formats (MEF one HDU for each detector, or one single-HDU fits for each detector). - New Butler dataset types bias/flat/fringe are added along with functions to standardize them to Exposures in {{DecamMapper}}. - Tests of retrieving bias/flat/fringe by Butler are added. - testdata_decam used by the unit tests is at lsst-dev:/lsst8/testdata_decam/ Known caveats for future users - The task could be a bit noisy when ingesting DECam calibration data products. But given the variety of data products on hand I might rather have those reminders than letting all pass silently. - I wouldn't be too surprised if future DECam Community-Pipeline products appear in different formats. Depending on how different they become, {{DecamCalibsParseTask}} might or might not need future modifications. ",21 DM-3768,"Resolve the issues found in the S15 end-to-end system exercise","There are a few items we need to take care to finish the end-to-end system for S15. ",8 DM-3769,"access the database created and populated for Bremerton end-to-end system","Collect the information for the tables populated for Bremerton end-to-end exercise. Use them in SUI/T so we can access them using the DAX API. ",2 DM-3770,"build the SUI system on NCSA to use the right database and tables","Due to the changes of the database and tables, the system has to be rebuilt.",1 DM-3771,"Resolve the issues accessing the newly populated tables","There are several issues need to be resolved for the system to work properly. ",5 DM-3772,"Fix compiler detection for non-default gcc/g++ compiler","{{scons CXX=g+\+-4.4}} launches {{g\+\+-4.4 --version}} which returns {{g++-4.4 (Debian 4.4.7-2) 4.4.7}}. Nevertheless the {{-4.4}} is not supported by Qserv compiler detection tool. Support will be added here",1 DM-3773,"add RUNID option to EventAppender","A RUNID needs to be added as an option to EventAppender to allow event logging selectors to receive only events for a particular run.",3 DM-3774,"lsst_build's default ref from repos.yaml support is broken when building multiple packages","A problem with the default ref in {{repos.yaml}} support implemented in DM-3679 was discovered last Friday, shortly after deploying this feature to the production CI systems. The default ref for {{xrootd}} was changed/overridden in {{repos.yaml}} to {{legacy/master}}. This worked as expected (and as was tested) when setting {{xrootd}} as the sole {{lsstswBuild.sh}} product or when running {{rebuild}} by hand. However, when building any package that pulled in {{xrootd}} as a recursive dependency, the {{master}} branch was being used (this case had not been manually tested).",1 DM-3775,"HSC backport: updates to tract and patch finding","This is a port of the following HSC updates to how tracts and patches are found and listed given a set of coordinates. These are all standalone commits (i.e. not associated with a ticket): [Add findTract() and findTractPatchList() in ringsSkyMap.|https://github.com/HyperSuprime-Cam/skymap/commit/761e915dde25ce8ed5622c2d84b83793e9580fd7] [move RingsSkyMap.findTractPatchList to BaseSkyMap.findClosestTractPatchlist|https://github.com/HyperSuprime-Cam/skymap/commit/56476142060bdb7d8c7fb59eacc383f0e0d5c85b] [Small bug fix for RingsSkyMap.findTract().|https://github.com/HyperSuprime-Cam/skymap/commit/f202a7780ebb89166f03479d7447ace1555027c1] [Add fast findTractPatchList() in RingsSkyMap.|https://github.com/HyperSuprime-Cam/skymap/commit/7e49c358501f95ce4c0e1aa8f48103a24391fc22] [Fixed the problems regarding poles and RA wrap.|https://github.com/HyperSuprime-Cam/skymap/commit/841b0c9eda7462a7a4f182b7971d5e8e81478bfe] [Add spaces around '+' and '-' to match LSST standard coding style.|https://github.com/HyperSuprime-Cam/skymap/commit/f7e2f036494afe382e653194c82bb15728c60fc3]",1 DM-3776,"LDM-144 Consistency Update","Due to the 2 year gap between the original authoring date of LDM-144 and the recent update, the 'costing only' update caused the document to be less self consistent than desired. This work is to do more than 'costing' updates such that the document is useful and on target to be more than a costing umbrella.",3 DM-3777,"server side preparation for histogram plot (2)","1. On the initialization, server needs to return the following summary table for all numeric columns: a. column name / description / unit b. min and max values c. number of points 2. For a given column and binning options, return the table of bins: a. first column - numInBin - number of points in a bin b. second column - binMin - bin's lower bound c. third column - binMax - bin's upper bound [10-13-15] Added by LZ Here is the detailed requirement from Tatiana: The implementation of this ticket are two search processors (similar to IpacTableFromSource search processor). Both search processors should be located in edu.caltech.ipac.firefly.server.query package and produce an IPAC Table. 1. Getting table statistics INPUT: The input to the first search processor should be TableServerRequest (treq) with ""searchRequest"" parameter set. The searchRequest parameter will be a serialized JSON string, which can represent a search request to any other search processor. This parameter determines how the input IPAC table is obtained. (The input IPAC table is produced similarly to how IpacTableFromSource produces the result with ""processor"" parameter set.) To parse SearchRequest parameter into key-value pairs - the parameters for the search request - you can use json.simple library: JSONObject searchRequestJSON = (JSONObject)JSONValue.parse(treq.get(""searchRequest"")); if (searchRequestJSON != null) { for (Object param : searchRequestJSON.keySet()) { String name = (String)param; String value = (String)searchRequestJSON.get(param); } } One of the parameters should be the ""id"" (ServerReuqest.ID_KEY) - which tells which search processor to use to obtain input IPAC table. OUTPUT: The output IPAC table produced by the first search processor should contain 5 columns: columnName, description, unit (unit or empty string), min, max, numPoints (the number of non-null values). 2. Getting histogram data INPUT: The input to the second processor should be TableServerRequest (treq) with ""searchRequest"" parameter set (exactly the same as ""searchRequest"" parameter in item 1) plus binningOptions. The binningOptions parameter will be serialized JSON object with the following keys (object properties): { ""algorithm"" : ""fixedSizeBins"", ""binSize"" : 20, ""column"" : ""wavelength"", ""scale"" : ""linear"", ""min"" : 0, ""max"" :100 } Notes: -the algorithm is simple fixed size binning for now, in future we'll need to add variable size bins algorithm described here: http://www.astroml.org/examples/algorithms/plot_bayesian_blocks.html - at the beginning, you can assume that column is column name, but it can be an expression based on several columns. - scale can be ""linear"" or ""log"" - min and max define filter on the points included into histogram calculation, any of them or both can be absent. If both are absent, no filter needs to be applied. OUTPUT: The output IPAC table produced by the second search processor should contain 3 columns: numInBin, binMin, binMax in that order. The table should be sorted by binMin. The binMax of row i should be always less or equal to binMin of row i+1. If number of points in the bin is 0, it's OK to skip this bin. ",12 DM-3778,"Fix compiler warns in protobuf clients","Google protobufs 2.6.1 includes a few unnecessary semicolons in some of its supplied header files; these generate a lot of compiler warnings when compiling client packages. Proposed fix is to add a patch to our eups t&p protobufs package to remove the offending semicolons.",1 DM-3779,"clean up gcc and eclipse code analyzer warns","We've been ignoring some accumulating warns in the qserv build for some time now. Now that it is possible to develop qserv in eclipse, it would be useful to address warns and analyzer issues so that we can start to notice when new ones pop up.",1 DM-3780,"Rationalize lsst/xrootd repo and maintenance procedures","The procedure for pulling/pushing xrootd changes from/to the upstream official xrootd repo is cumbersome, confusing, and error-prone. Buildbot now has support for releasing packages from branches other than master. Given this, we can now reasonably replace our lsst/xrootd repo with a fresh genuine fork (shared history) of upstream, then carry our lsst-specific work forward on a dev-branch. This will make it much easier to track and contribute to the xrootd project moving forward. Existing legacy branches and tags are to be migrated to the fresh fork, so historical builds will not be broken.",1 DM-3782,"Review, plan, procure development infrastructure (FY15)",NULL,3 DM-3783,"Refining file system policies",NULL,2 DM-3784,"W16 Operation of Joint Coordination Council","Activities associated with implementation of the MOA establishing CC-IN2P3 as a satellite processing center during operations. Includes preparation and execution of monthly JCC meetings, as well as a face-to-face meeting for intense coordination scheduled for early November. Assignees: Don Petravick, Margaret Gelman, Jason Alt, Robert Gruendl Duration: September 2015 - February 2016",56 DM-3785,"W16 processing control emergent work","Bucket epic for bug fixes and *minor* work that falls outside planned epics. Assignees: Steve Pietrowicz, Greg Daues, Matias Carrasco Kind, Hsin-Fang Chiang, James Parsons Duration: September 2015 - February 2016",30 DM-3786,"Update sizing model for February 2016 refresh - technology and costing","Biannual refresh of Sizing Model (LDM-144 et al.), including updates to both costing and technology. As it will be the first technology refresh in at least 2 years, we anticipate it will take a considerable amount of time. To prepare and gain insight on technology and costing trends, this activity includes attending SC2015 in mid-Nov. Assignees: Jason Alt Duration: November 2015 - January 2015",38 DM-3787,"Liaise all groups to commission OpenStack for LSST","Greg has been appointed the service manager of the NCSA nebula for LSST. Work with LSST developers and NCSA system engineers to commission the OpenStack for LSST's use. Assignees: Greg Daues Duration: September 2015 - February 2016",47 DM-3788,"FY16 Hardware Purchasing Plan","The Annual Acquisition Strategy Document describes the capabilities (hardware, compute cycles, software, licenses, etc.) planned for procurement during the fiscal year. We consider systems that will satisfy the needs of developers, systems for prototyping the production infrastructure, etc. Assignees: Jason Alt, Bill Glick Duration: September - October 2015",11 DM-3791,"Evaluate PASTRY DHT implementation","The David Keller kademlia implementation used in the earlier prototype has some bugs/limitations. Try to find a better off-the-shelf DHT and integrate with prototype framework.",6 DM-3792,"obs_test data mis-assembled","obs_test images are mis-assembled and need to be regenerated. This may affect some existing unit tests that rely on the data.",2 DM-3794,"(FY16) Initial discussions and requirement consolidation",NULL,2 DM-3796,"remove install_name_tool fix to libpython2.7.dylib from anaconda package","Now that SIM-1314 has been merged, we should be able to remove the {code} if [[ $(uname -s) = Darwin* ]]; then #run install_name_tool on all of the libpythonX.X.dylib dynamic #libraries in anaconda for entry in $PREFIX/lib/libpython*.dylib do install_name_tool -id $entry $entry done fi {code} from eupspkg.cfg.sh in the EUPS anaconda package, and still have GalSim build correctly",1 DM-3797,"Enable SSL to community.lsst.org","Enable SSL (https) for the Discourse site at community.lsst.org",1 DM-3798,"Update flag names and config override files to current conventions","The {{deblend.masked}} and {{deblend.blendedness}} flag names in {{meas_deblender}} need to be updated to use underscores instead of periods. Various flag names in the {{examples}} scripts also need updating to the underscore and camelCase format. A search for these flags throughout the database revealed a number of config files that need updating to current conventions. These are also included here.",1 DM-3800,"testProcessCcd.py computes values that are too different between MacOS and linux","tests/testProcessCcd.py runs processCcd on visit 1 of obs_test's data repository. The result on MacOS is surprisingly different than on linux in at least one case: psfShape.getIxx() computes 2.71 on MacOS X and 2.65 on linux. Iyy and Ixy are likely different. It's worth checking all other computed values, as well. These differences likely indicate that something is wrong, e.g. in obs_test, processCcd, or the way the test runs processCcd. This showed up as part of fixing DM-3792, but it is not clear if the changes on DM-3792 actually caused or increased the difference between MacOS and linux, or if the difference was always too large, but was masked by an intentionally generous tolerance in the unit test.",2 DM-3801,"The gains in obs_test's amplifier table appear to be incorrect","As of DM-3792 the gains in obs_test's camera's amplifier table were set to the values reported in the headers of the lsstSim raw data used to generate obs_test's raw data. (Before that one nominal gain was used for all amplifiers). However, [~rhl] reports that these gains are incorrect. He measured the following gains by scaling the nominal gains by the median values in the bias-subtracted data: {code} amp meas curr name gain gain 00 1.7741 1.7741 01 1.8998 1.65881 10 1.8130 1.74151 11 1.8903 1.67073 {code} We could simply adopt these values, but I would like to understand why the gains are so far off from those reported by phoSim in the raw data.",4 DM-3802,"The obs_test's sensor is shown 90 degrees rotated from that desired, in camera coords","When plotting the obs_test sensor, e.g. using lsst.afw.cameraGeom.utils.plotFocalPlane, the image is a short, wide rectangle. This suggests that the camera coordinate frame is rotated 90 degrees from the CCD coordinate frame (which has 1018 pixels in X and 2000 pixels in Y). We would prefer that the camera frame and CCD frame have the same orientation.",1 DM-3803,"Fix Qserv compiler warnings with clang","Qserv triggers numerous warnings with clang on OS X. Full details are in the attached ticket, here we summarize the distinct warnings classes: h5. Protobuf {code} /Users/timj/work/lsstsw/stack/DarwinX86/protobuf/2.6.1+fbf04ba888/include/google/protobuf/unknown_field_set.h:214:13: warning: anonymous types declared in an anonymous union are an extension [-Wnested-anon-types] mutable union { ^ {code} h5. Qserv {code} In file included from core/modules/sql/statement.cc:32: core/modules/sql/Schema.h:74:1: warning: 'Schema' defined as a struct here but previously declared as a class [-Wmismatched-tags] struct Schema { ^ core/modules/sql/statement.h:35:1: note: did you mean struct here? class Schema; // Forward ^~~~~ struct {code} {code} core/modules/proto/WorkerResponse.h:34:1: warning: 'WorkerResponse' defined as a struct here but previously declared as a class [-Wmismatched-tags] struct WorkerResponse { ^ core/modules/ccontrol/MergingRequester.h:38:3: note: did you mean struct here? class WorkerResponse; ^~~~~ struct {code} {code} In file included from core/modules/qana/QueryMapping.cc:46: core/modules/qproc/ChunkSpec.h:51:1: warning: 'ChunkSpec' defined as a struct here but previously declared as a class [-Wmismatched-tags] struct ChunkSpec { ^ core/modules/qana/QueryMapping.h:44:5: note: did you mean struct here? class ChunkSpec; ^~~~~ struct {code} {code} core/modules/qana/TableInfo.h:186:1: warning: 'DirTableInfo' defined as a struct here but previously declared as a class [-Wmismatched-tags] struct DirTableInfo : TableInfo { ^ core/modules/qana/TableInfo.h:86:1: note: did you mean struct here? class DirTableInfo; ^~~~~ struct core/modules/qana/TableInfo.h:221:1: warning: 'ChildTableInfo' defined as a struct here but previously declared as a class [-Wmismatched-tags] struct ChildTableInfo : TableInfo { ^ core/modules/qana/TableInfo.h:87:1: note: did you mean struct here? class ChildTableInfo; ^~~~~ struct core/modules/qana/TableInfo.h:260:1: warning: 'MatchTableInfo' defined as a struct here but previously declared as a class [-Wmismatched-tags] struct MatchTableInfo : TableInfo { ^ core/modules/qana/TableInfo.h:88:1: note: did you mean struct here? class MatchTableInfo; ^~~~~ struct In file included from core/modules/qana/ColumnVertexMap.cc:36: core/modules/qana/RelationGraph.h:513:1: warning: struct 'Vertex' was previously declared as a class [-Wmismatched-tags] struct Vertex; ^ core/modules/qana/ColumnVertexMap.h:44:7: note: previous use is here class Vertex; ^ In file included from core/modules/qana/ColumnVertexMap.cc:36: core/modules/qana/RelationGraph.h:547:1: warning: 'Vertex' defined as a struct here but previously declared as a class [-Wmismatched-tags] struct Vertex { ^ core/modules/qana/ColumnVertexMap.h:44:1: note: did you mean struct here? class Vertex; ^~~~~ struct {code} {code} core/modules/wbase/Base.h:72:1: warning: 'ScriptMeta' defined as a struct here but previously declared as a class [-Wmismatched-tags] struct ScriptMeta { ^ core/modules/wbase/Task.h:41:5: note: did you mean struct here? class ScriptMeta; ^~~~~ struct {code} {code} In file included from core/modules/parser/BoolTermFactory.cc:46: core/modules/query/Predicate.h:86:27: warning: 'lsst::qserv::query::GenericPredicate::putStream' hides overloaded virtual function [-Woverloaded-virtual] virtual std::ostream& putStream(std::ostream& os) = 0; ^ core/modules/query/Predicate.h:71:27: note: hidden overloaded virtual function 'lsst::qserv::query::Predicate::putStream' declared here: different qualifiers (const vs none) virtual std::ostream& putStream(std::ostream& os) const = 0; ^ {code} {code} core/modules/parser/FromFactory.cc:62:15: warning: unused function 'walkToSiblingBefore' [-Wunused-function] inline RefAST walkToSiblingBefore(RefAST node, int typeId) { ^ core/modules/parser/FromFactory.cc:72:1: warning: unused function 'getSiblingStringBounded' [-Wunused-function] getSiblingStringBounded(RefAST left, RefAST right) { ^ {code} {code} In file included from core/modules/wsched/ChunkDisk.cc:25: core/modules/wsched/ChunkDisk.h:130:10: warning: private field '_completed' is not used [-Wunused-private-field] bool _completed; ^ {code} {code} In file included from core/modules/parser/PredicateFactory.cc:45: core/modules/query/Predicate.h:86:27: warning: 'lsst::qserv::query::GenericPredicate::putStream' hides overloaded virtual function [-Woverloaded-virtual] virtual std::ostream& putStream(std::ostream& os) = 0; ^ core/modules/query/Predicate.h:71:27: note: hidden overloaded virtual function 'lsst::qserv::query::Predicate::putStream' declared here: different qualifiers (const vs none) virtual std::ostream& putStream(std::ostream& os) const = 0; ^ {code} {code} core/modules/parser/WhereFactory.cc:265:31: warning: binding reference member 'c' to stack allocated parameter 'c_' [-Wdangling-field] PrintExcept(Check c_) : c(c_) {} ^~ core/modules/parser/WhereFactory.cc:291:28: note: in instantiation of member function 'lsst::qserv::parser::PrintExcept::PrintExcept' requested here PrintExcept p(mc); ^ core/modules/parser/WhereFactory.cc:269:12: note: reference member declared here Check& c; ^ {code} {code} core/modules/rproc/ProtoRowBuffer.cc:44:11: warning: unused variable 'largeRowThreshold' [-Wunused-const-variable] int const largeRowThreshold = 500*1024; ^ {code} {code} core/modules/util/testIterableFormatter.cc:85:43: warning: suggest braces around initialization of subobject [-Wmissing-braces] std::array iterable { ""1"", ""2"", ""3"", ""4"", ""5"", ""6""}; ^~~~~~~~~~~~~~~~~~~~~~~~~~~~ { } {code} {code} In file included from core/modules/qdisp/XrdSsiMocks.cc:37: core/modules/qdisp/XrdSsiMocks.h:64:16: warning: private field '_executive' is not used [-Wunused-private-field] Executive *_executive; ^ {code} {code} core/modules/xrdoss/QservOss.cc:77:1: warning: unused function 'print' [-Wunused-function] print(std::ostream& os, lsst::qserv::xrdoss::QservOss::StringSet const& h) { ^ {code} h5. OS X {code} core/modules/qdisp/QueryRequest.h:54:25: warning: 'lsst::qserv::qdisp::BadResponseError::what' hides overloaded virtual function [-Woverloaded-virtual] virtual char const* what() throw() { ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/exception:95:25: note: hidden overloaded virtual function 'std::exception::what' declared here: different qualifiers (const vs none) virtual const char* what() const _NOEXCEPT; ^ In file included from core/modules/qdisp/Executive.cc:64: In file included from core/modules/qdisp/XrdSsiMocks.h:37: core/modules/qdisp/QueryRequest.h:67:25: warning: 'lsst::qserv::qdisp::RequestError::what' hides overloaded virtual function [-Woverloaded-virtual] virtual char const* what() throw() { ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/exception:95:25: note: hidden overloaded virtual function 'std::exception::what' declared here: different qualifiers (const vs none) virtual const char* what() const _NOEXCEPT; ^ {code} {code} core/modules/proto/TaskMsgDigest.cc:55:5: warning: 'MD5' is deprecated: first deprecated in OS X 10.7 [-Wdeprecated-declarations] MD5(reinterpret_cast(str.data()), ^ /usr/include/openssl/md5.h:116:16: note: 'MD5' has been explicitly marked deprecated here unsigned char *MD5(const unsigned char *d, size_t n, unsigned char *md) DEPRECATED_IN_MAC_OS_X_VERSION_10_7_AND_LATER; ^ {code} {code} core/modules/util/StringHash.cc:78:24: warning: 'SHA1' is deprecated: first deprecated in OS X 10.7 [-Wdeprecated-declarations] return wrapHashHex(buffer, bufferSize); ^ /usr/include/openssl/sha.h:124:16: note: 'SHA1' has been explicitly marked deprecated here unsigned char *SHA1(const unsigned char *d, size_t n, unsigned char *md) DEPRECATED_IN_MAC_OS_X_VERSION_10_7_AND_LATER; ^ core/modules/util/StringHash.cc:83:24: warning: 'SHA256' is deprecated: first deprecated in OS X 10.7 [-Wdeprecated-declarations] return wrapHashHex(buffer, bufferSize); ^ /usr/include/openssl/sha.h:150:16: note: 'SHA256' has been explicitly marked deprecated here unsigned char *SHA256(const unsigned char *d, size_t n,unsigned char *md) DEPRECATED_IN_MAC_OS_X_VERSION_10_7_AND_LATER; ^ {code} h5. Xrootd {code} In file included from core/modules/qdisp/Executive.cc:64: In file included from core/modules/qdisp/XrdSsiMocks.h:33: In file included from /Users/timj/work/lsstsw/stack/DarwinX86/xrootd/u.timj.DM-3584-ge22410fa7f+da39a3ee5e/include/xrootd/XrdSsi/XrdSsiRequest.hh:37: /Users/timj/work/lsstsw/stack/DarwinX86/xrootd/u.timj.DM-3584-ge22410fa7f+da39a3ee5e/include/xrootd/XrdSsi/XrdSsiRespInfo.hh:43:1: warning: 'XrdSsiRespInfo' defined as a struct here but previously declared as a class [-Wmismatched-tags] struct XrdSsiRespInfo ^ /Users/timj/work/lsstsw/stack/DarwinX86/xrootd/u.timj.DM-3584-ge22410fa7f+da39a3ee5e/include/xrootd/XrdSsi/XrdSsiSession.hh:45:1: note: did you mean struct here? class XrdSsiRespInfo; ^~~~~ struct {code} {code} core/modules/xrdoss/QservOss.h:64:17: warning: 'lsst::qserv::xrdoss::FakeOssDf::Opendir' hides overloaded virtual function [-Woverloaded-virtual] virtual int Opendir(const char *) { return XrdOssOK; } ^ /Users/timj/work/lsstsw/stack/DarwinX86/xrootd/u.timj.DM-3584-ge22410fa7f+da39a3ee5e/include/xrootd/XrdOss/XrdOss.hh:63:17: note: hidden overloaded virtual function 'XrdOssDF::Opendir' declared here: different number of parameters (2 vs 1) virtual int Opendir(const char *, XrdOucEnv &) {return -ENOTDIR;} ^ {code} {code} In file included from core/modules/xrdsvc/SsiSession.h:32: /Users/timj/work/lsstsw/stack/DarwinX86/xrootd/u.timj.DM-3584-ge22410fa7f+da39a3ee5e/include/xrootd/XrdSsi/XrdSsiResponder.hh:177:27: warning: control may reach end of non-void function [-Wreturn-type] } ^ {code} h5. boost {code} /Users/timj/work/lsstsw/stack/DarwinX86/boost/1.55.0.1.lsst2+fbf04ba888/include/boost/regex/v4/regex_raw_buffer.hpp:132:7: warning: 'register' storage class specifier is deprecated [-Wdeprecated-register] register pointer result = end; ^~~~~~~~~ {code}",1 DM-3804,"Fix order of arguments change in meas_base SingleFrameMeasurement","In sfm.py on line 271, a comment indicates that some code is a temporary work around until the switch from meas_algorithms to meas_base is complete. This work is complete, so this temporary workaround should be removed, or if it is decided it should be kept, the comment should be removed. See https://github.com/lsst/meas_base/blob/tickets/DM-2915/python/lsst/meas/base/sfm.py#L271",2 DM-3806,"convert newinstall.sh to use miniconda instead of anaconda","To match the conversion of lsstsw from anaconda -> miniconda to reduce the disk footprint and improve install times.",1 DM-3808,"Setup lsst_sphinx_kit package structure","Setup the lsst_sphinx_kit package, including * setup.py * unit tests, tox and Travis CI * README stub * Sphinx stub and readthedocs",1 DM-3811,"HSC backport: Include documentation strings for config parameters when they are dumped","This is a port of the following HSC tickets: [HSC-1072|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1072] and [HSC-1175|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1175]",4 DM-3814,"Migrate LDM-152 to reST Design Doc Platform",NULL,1 DM-3815,"Intermittent build failures on v11 candidate with eups distrib","We are seeing frequent intermittent failures on a variety of platforms when installing the v11 candidate with eups distrib install. Repeating the command works. It doesn't seem to be evenly distributed between packages: cfitsio, -meas_astrom-(?), meas_algorithms, and obs_lsstSim have been seen multiple times. It's been seen in the release verification CI (that uses the documented user facing process of eups distrib install instead of the factory CI which uses lsstsw) and in individual user ""manual"" builds on lsst-dev. Test build key: ||buildbot build # || eups tag || refs || comment || ||b1688 || t20150914-b1688 || tickets/DM-3829 v11_0_rc2 || || ||b1689 || t20150914-b1689 || tickets/DM-3829 tickets/DM-3815-intermittent-build-failure v11_0_rc2 || || ||b1690 || t20150914-b1690 || tickets/DM-3829 v11_0_rc2 master || modifications to tickets/DM-3829 || ||b1692 || t20150915-b1692 || tickets/DM-3829 ticket/DM-2752-egg-error v11_0_rc2 || ||",6 DM-3816,"levels in DecamMapper.paf is not quite right","When ccdnum is not given as part of the dataId, instead of iterating over it, an error like this happens {code:java} RuntimeError: No unique lookup for ['ccdnum'] from {'visit': 205344}: 61 matches {code} Likely a problem in policy/DecamMapper.paf",1 DM-3817,"Deploy docker images on ccqserv124/149","Creation of configured master and worker image will be improved here, and a deploymen tool (like swarm, of hand-made) will be used to deploy images over in2p3 cluster.",6 DM-3818,"Bi-weekly PO security meeting","Bi-weekly meeting with PO on cyber security.",1 DM-3819,"IdM work","Work for LSST Identity management and authentication.",2 DM-3820,"NIST SP 800.82 investigation","NIST SP 800.82 investigation for a more cohesive SCADA enclave security plan.",2 DM-3821,"Recent CModel bugfixes from HSC","I've just fixed two rather critical bugs in the CModel code on the HSC side (they would have been introduced on the LSST side in the last transfer, DM-2977): - The {{minInitialRadius}} configuration parameter had a default that is too small, causing many galaxies to be fit with point source models, leading to bad star/galaxy classifications. This is HSC-1306. - There was a simple but important algebra error in the uncertainty calculation, making the uncertainty a strong function of magnitude. This is HSC-1313. On the LSST side, the transfer should be quite simple; we'll have to rewrite a bit of code due to the difference in measurement frameworks, but there was very little to begin with (most of the effort in the HSC issues was in debugging).",1 DM-3822,"Orchestration work to support verification dataset processing","Upgrades, modifications, fixes, etc. to orchestration framework for use in SQuaRE's verification dataset processing tests. Assignees: Steve Pietrowicz, Greg Daues Duration: September 2015 - February 2016",27 DM-3823,"W16 General Management Activities",NULL,100 DM-3824,"meas_astrom bugs exposed by new Eigen","Trying a newer Eigen has exposed several issues in meas_astrom: - tests/createWcsWithSip.py blindly uses sipWcs in the result returned by ANetBasicAstrometryTask.determineWcs2, but that attribute may be None - ANetBasicAstrometryTask.determineWcs2 terminates iteration early if the # of matches goes down, even though the result may be improved. In the case in question the first fit iteration results in significantly better RMS error, but has one fewer matches, so the SIP fit is rejected, triggering the first bug mentioned.",6 DM-3825,"write meeting agenda, for Sept 14 meeting ",NULL,1 DM-3826,"Further refinement ","added the image and engineering facility database. and Observatory Operations Server.",10 DM-3827,"email discussion w.r.t Service separation for L1, and also some work on ITIL roles","email to German about the all the L1 stuff and clarified that the L1 system were a derive provided to the Telescopes site (important for fitting this into the proper place in the who is worrying about what hierarchy. Also, revised SA to see fi the EPO changes discusses affected the IT roles in th model (not apparently) lastly attitude the TOWC ",2 DM-3828,"Management for week ending sept 11","Deal with Hiring James Parsons, and interfacing with the new NCSA organizations that will support LSST at NCSA, general group management issues",2 DM-3830,"Deploy FY16 Storage Expansion","Install, and deploy a prototype production GPFS cluster.",20 DM-3831,"Preparation work to process raw DECam data","Try to run processCcd.py with raw DECam data and see what are yet to be solved for it to run. ",7 DM-3832,"Deploy FY16 Nebula Expansion","* Install, test, and deploy additional capacity for the NCSA nebula. (Note: this work occurs outside of LSST so we can only track it at a less granular detail than other deployments. We can include the final story for blockers however.)",18 DM-3833,"Decommission old development infrastructure","Decommission old hardware currently in the LSST development cluster when replacement equipment arrives and is provisioned. Assignees: Bill Glick, Matt Elliot, Bruce Mather Duration: November - December 2015",19 DM-3834,"Migrate LDM-230 to new docs platform","Convert LDM-230 from Word to restructuredText and deploy on readthedocs.org",2 DM-3835,"Migrate LDM-135 to new design docs platform","Convert LDM-135 from Word to restructuredText and deploy on readthedocs.org",5 DM-3836,"Migrate LDM-129 to new design docs platform","Convert LDM-129 from Word to restructuredText and deploy onto readthedocs.org",2 DM-3837,"SuperTask Redesign","Redesign pipe_base to allow the creation of supertasks which will be more flexible for different execution applications",11 DM-3838,"Evaluate authentication and authorization services for user workspace","Develop an Identity and Access Management (IAM) program. Deliverables (from SOW): - LSST IAM Design Document: describe LSST's current and expected IAM needs and specify technical recommendations for the LSST IAM system architecture, including interface standards (e.g., LDAP, OAuth) and system components (with functional descriptions and implementation recommendations). - LSST IAM Program Plan: specify future and ongoing IAM development and operational activities required to meet the LSST project mission. - LSST IAM Technical Demonstration: provide an initial implementation of the LSST IAM design according to the philosophy of ""rough consensus and running code."" The project will prioritize technical implementation work based on the immediate needs identified by LSST developers for functional implementations of IAM system interfaces to ""fill the gaps"" needed for LSST development and integration work to proceed on schedule. Assignees: Jim Basney, Terry Fleurry, Daniel Thayer, Alex Withers (ISO), Don Petravick, Jason Alt, Margaret Gelman Duration: October 2015 - March 2016",88 DM-3839,"Graphical representation Example demo",NULL,2 DM-3840,"LSE-72: Phase 3 in X16","Advance Phase 3 details as needed to eliminate obstacles to OCS and DM development during F16.",8 DM-3841,"LSE-75: Refine WCS and PSF requirements in W16","Clarify the data format and precision requirements of the TCS (or other Telescope and Site components) on the reporting of WCS and PSF information by DM on a per-image basis. Depends on the ability of the T&S group to engage with this subject. Current PMCS deadline for Phase 3 readiness of LSE-75 is 29-Sep-2015.",8 DM-3842,"LSE-68: ICD Details in X16","Bring ICD to phase 3 level of detail",6 DM-3843,"LSE-74: ICD Details in W16","""Bring ICD to phase 3 level of detail"" was the original specification, but actual work by the OCS group in the Winter 2016 period didn't quite reach Phase 3. Nonetheless, a very useful revision was submitted to and recommended for approval by the CCB at the March meeting.",6 DM-3844,"Add tutorial-level documentation for ctrl_pool","The new ctrl_pool package (port of hscPipeBase) has no tutorial-level documentation, making it hard to figure out how to start using the package. Unfortunately, I think only [~price] is qualified to write it directly, though it may make sense to have someone unfamiliar write it while bugging Paul a lot, both to transfer the knowledge and target the documentation better.",4 DM-3845,"Add unit tests for ctrl_pool","ctrl_pool (formerly hscPipeBase) is being ported with no unit tests - the only testing is an example script that can be run by hand to demonstrate a piece of the functionality. Some functionality may simple not be amenable to tests (such as batch submission). Other parts may be tricky to run via SCons because they're intrinsically parallel, and SCons naively wants to be able to run each test in a separate process. Overcoming those problems is the reason this is challenging - there isn't really that much functionality to test.",8 DM-3846,"Read over LSE-70 and LSE-209 and discuss for meeting","Read over LSE-70 and LSE-209 for meeting on Friday 9/11.",2 DM-3847,"Grid overly bug","when using the grid overlay in galatic coordinate over an image that is around longitude = 0, the overlay doesn't work properly. In BOLOCAM, on fits image works but not the other one. Reproducible: Steps to Reproduce: Go to Atlas search, and select BOLOCAM galactic plane survey (GPS). Then enter single coordinate search on ""0 0 gal"" to find images taken around that. See column ""FITS filename"" and search for the sharc-ii instrument images such as images/sharc2/L000.15+0.00.fits [bad] and images/sharc2/L000.00+0.00.fits [good] Then open it on irsaviewer by clicking on the icon link (first column). On the image viewer, enable the grid overlay and click on the icon 'layer' on the toolbar. Change the coordinate system to galactic to see the problem. On the overlay for the associated Bolocam map there are three horizontal lines which exist only on the left hand side of the image. When the image coordinate system is set to ""Gal"", the reported GLON values range from -355.4 to -1.6 (from left to right), passing through -0.0 in the center. Same problem can be seen for images/v1/INNER_GALAXY/map/v1.0.2_super_gc_13pca_map50_crop.fits [bad] ",4 DM-3849,"evaluate NCSA OpenStack against SQRE requirements and provide feedback - part 2",NULL,1 DM-3850,"Nebula metadata service is intermittent","Upon restarting one of my nebula instances (ktl-test), I noticed a failure in the logs: {quote} Sep 14 18:07:29 ktl-test cloud-init: 2015-09-14 18:07:29,157 - util.py[WARNING]: Failed fetching metadata from url http://169.254.169.254/latest/meta-data {quote} Attempting to retrieve that URL seems to randomly vary between succeeding, which returns: {quote} ami-id ami-launch-index ami-manifest-path [...] {quote} and failing, which returns: {quote} 500 Internal Server Error

500 Internal Server Error

Remote metadata server experienced an internal server error.

{quote} These failures may be contributing to observed sporadic {{ssh}} key injection failures.",2 DM-3851,"Liaise with Long-Haul Network group on Base Site to NCSA network","Coordinate with network group to establish Base Site-to-NCSA LHN. Assignees: Paul Wefel Duration: September 2015 - February 2016",4 DM-3852,"Evaluate file management technologies","Evaluate technologies for data management (e.g., iRODS).",100 DM-3853,"Consult on contracting and execution of Chilean Data Center contract","Continue to consult and review design documents for the construction of the Chilean DAC. Assignees: Tom Durbin Duration: September 2015 - February 2016",11 DM-3855,"Calibration products preparation","Develop the calibration products pipeline plan and begin initial implementation.",22 DM-3857,"Stack documentation Part I","Original documentation based on expressed team needs. [JS 100%] ",30 DM-3858," Integration and test monitoring architecture Part II","Work to be assessed on the basis of the outcome of Part I [DM-2050] [100% JMP] ",70 DM-3859,"Supertask design and prototyping - Part I","This epic covers design and prototyping work for an encapsulation of tasks that allows for the chaining and interleaving of tasks with QA/metrics tasks, intergration tests and/or KPM/analysis afterburners. The emphasis at this point is on speed of development and customisation; performance is assumed not to be an issue as it will eventually be guaranteed by the process execution framework. Only SQuaRE's effort is covered by this epic; additional effort on this investigation is provided by the Middleware WBS and not captured here. The outcome of this epic is a proposal for moving forward. In the event that it is acceptable to the Project Engineer, further epics might be define, hence Part I. [100% GPDF] ",42 DM-3860,"Communication Toolchain support","This epic covers support of communication tools primarily used by DM and/or supported by DM on behalf of other parts of the project - JIRA, Discourse, Hipchat, etc The source of this work is primarily driven by short-term user requests, and so the outcome is timeboxed rather than planned. [JS 50% FE 50%] ",20 DM-3861,"QA Architecture Documents","Design document outlining SQuaSH elements and implementation plan. [FE 100%]",10 DM-3862,"Stack Build, Packaging and Testing Improvements Part II","This epic is an umbrella for RFC-69 work. [JH 90% JS 10%] ",60 DM-3863,"Web design fixes DM Design Documents on Sphinx/Read The Docs","Solve fit-and-finish issues with the stock readthedocs.org Sphinx template when rendering DM design documents. Issues include: * Sections need to be numbered and those numbers need to appear in TOC * RTD's TOC does not properly collapse sub-topics * Appropriate styling for document title and author list * Wrapping the changelog table * Adapt section references so that just the section number can be referenced, independently of the section number and title in combination * Section labels given explicitly in the reST markup are different from the anchors that Sphinx gives to the {{}}tags; the former are simply divs inserted in the HTML. The solutions may involve # reconfiguring the Sphinx installation of individual documents # forking the RTD HTML template, and/or # developing extensions for Sphinx in {{sphinxkit}}.",2 DM-3864," Integration Dataset for metrics and regression tests - Part I "," This epic covers the defintion of a compact but rich dataset to support regression testing and surrogate metric development for regular automatic integration tests. It also involes a definition of interim/surrogate metrics where they can aid testing and development and/or where KMPs cannot be calculated. [DN 75% MWV 25%]",50 DM-3865," Processing of DECAM and other Verification Datasets","This epic covers work to lead and co-ordinate the processing of precursor datasets, DECAM in the first instance, through the stack and produce an assessmment of progress and preliminary metrics. In W16 execution will be done using the orca/HTCondor setup previously used at NCSA for Data Challenges. [DN 50% AF 25% JS 25%] ",100 DM-3866,"Do basic tests of CModel ellipticity measurements","I have seen enough anomalies that I have had to go back a bit and do some basic tests of CModel and its ellipticity bias. This involves running the same pipeline as before, but with controlled galaxy profiles, ellipticities, and angles. These are zero-shear, zero-seeing tests which I probably should have run first thing. It I understand everything I see in these tests, I will be confident that the results I am seeing for CModel with varying footprint size, varying nInitialRadii, and varying stamp size are correct.",4 DM-3869,"Simultaneous astrometry requirements","The HSC group needs improvements to the simultaneous astrometry fitter delivered by Astier et al. In particular we'll need to do simultaneous photometry as well. This epic is to determine the superset requirements for such a system.",9 DM-3870,"Refactor Jointcal to use stack functionality","Jointcal currently has a lot of built-in features that already exist elsewhere in the stack (e.g. GTransfo, StarSelector, Points, etc.). These features should be removed and replaced with the equivalent stack functionality.",36 DM-3871,"Implement simultaneous photometry","The simultaneous astrometry framework should be able to be extended to also fit the photometry at the same time. This task is to do just that. The task to create a pluggable framework should help with this.",56 DM-3872,"Clean up Wcs classes","The Wcs classes as they currently exist are not easy to extend and also contains overrides that are a bit ad hoc. This task is to clean up the existing classes so that there is a single abstract base class. It should also be a priority of this task to determine whether an upgrade in wcslib helps with the special cases (e.g. TAN-SIP).",38 DM-3873,"Gather requirements for improved Wcs classes","The new Wcs class should be able to apply transforms in a stack so that many different distortions as different scales can be corrected for separately (rather than trying to correct the whole mess with a single 2D polynomial). There is some work going on in the community around this. This task should include conversations with community leaders in this area.",19 DM-3874,"Produce a design for the new Wcs classes","This will require generating a design as well as getting it reviewed via RFC. The hope is that other work can be ongoing while the RFC process is carried out.",50 DM-3875,"Implement new Wcs classes","Once a design is accepted implement the new design. This epic covers the replacement of afw:image:wcs with afw:transform/afw:mapping (names currently in flux), but does not involve changes to XYTransform.",39 DM-3876,"Make Wcs persistable","Regardless of the mechanism used by the new Wcs classes, they will need to be persistable. This will probably require some significant work unless classes that are already persistable are used in the design.",56 DM-3877,"Identify all corrections ISR needs to handle","This is informed by DECAM etc. This is just for the corrections that do not require detection. For example, we will push out CR rejection until the question of snaps is decided.",9 DM-3878,"Implement all ISR corrections for LSST","Most corrections have some implementation. Perhaps the most difficult will be crosstalk since the implementation should allow for correction over multiple chips.",19 DM-3879,"Make up a test for dipole measurement","To facilitate work on the dipole measurement, a test will be helpful. This could simply be an image with dipoles made with Gaussian PSFs with Poisson noise on top. The important thing is to be able to determine whether the algorithm is returning the right answer.",9 DM-3880,"Improve the dipole measurement task","The current dipole measurement algorithm has some issues. It also doesn't work in the current measurement framework. This task is to improve the dipole measurement. The result of this epic should be improved results when run on the test data.",38 DM-3881,"Gather requirements to inform a redesign of the CalibrateTask","The current calibrate task is fairly brittle and hard to extend. This task is to gather the necessary requirements for a redesigned calibrate task.",2 DM-3882,"Implement the new CalibrateTask design","Implement the redesigned CalibrateTask.",28 DM-3883,"Create initial cluster design, send internally for feedback and planning","Gather feedback on initial designs for FY16 purchase plans.",2 DM-3884,"Create data products description","Addition of [https://confluence.lsstcorp.org/display/~petravick/Products+of+Image+Ingest+and+Processing] to understand more of the requirements necessary for the functional design ",3 DM-3885,"LSE-78: W16 revisions, harmonization with existing design","Review LSE-78 for self-consistency and consistency with the current DM and overall system design.",6 DM-3886,"Revise early integration milestones, LCR-323 and beyond","Revise the list of early integration milestones with OCS, TCS, CCS, and DAQ to form a coherent plan. Coordinate with NCSA and other interested parties in DM.",6 DM-3887,"Review ICD flowdown to DMSR and design documents",NULL,8 DM-3888,"Add missing space after if in Qserv code to conform to standard","Replace ""if("" with ""if ("" to follow standard. find core/modules/ -name ""*.cc"" |xargs grep ""if(""|wc -l 852 ",1 DM-3891,"Review LCR-323 proposal for integration milestones","Prepare for CCB action on LCR-323. Ensure that DAQ integration is included (it's not in the original LCR proposal).",4 DM-3892,"Review current version of LSE-78, prepare for LCR","Do a comprehensive read-through of the previous released version of LSE-78. Look for self-consistency and for consistency with the rest of the DM and overall system design. Report issues to appropriate people.",3 DM-3893,"Research existing DHT-based FS approaches ","The previous prototype provided confidence that a DHT overlay could work for routing and placement at the scales and time constants needed for chunk distribution. The next prototype will need actual data transport and storage management facilities layered on the node/key management provided by the DHT layer. Explore existing works at this level to a greater depth pick from among proven approaches.",6 DM-3894,"Provide values for relative astrometry KPMs in FY15","Should produce the numeric values required (or an explanation of why they aren't available) together with a description of the process for generating them (incl. the data processed, scripts used for plotting, etc).",2 DM-3895,"Provide values for PSF ellipticity KPMs in FY15","Should produce the numeric values required (or an explanation of why they aren't available) together with a description of the process for generating them (incl. the data processed, scripts used for plotting, etc).",1 DM-3896,"Provide values for photometric repeatability KPMs in FY15"," Should produce the numeric values required (or an explanation of why they aren't available) together with a description of the process for generating them (incl. the data processed, scripts used for plotting, etc).",4 DM-3897,"Provide value for DRP computational budget KPM in FY15","Should produce the numeric values required (or an explanation of why they aren't available) together with a description of the process for generating them (incl. the data processed, scripts used for plotting, etc).",3 DM-3898,"Fix xrootd compiler warnings with clang","h5. Xrootd {code} In file included from core/modules/qdisp/Executive.cc:64: In file included from core/modules/qdisp/XrdSsiMocks.h:33: In file included from /Users/timj/work/lsstsw/stack/DarwinX86/xrootd/u.timj.DM-3584-ge22410fa7f+da39a3ee5e/include/xrootd/XrdSsi/XrdSsiRequest.hh:37: /Users/timj/work/lsstsw/stack/DarwinX86/xrootd/u.timj.DM-3584-ge22410fa7f+da39a3ee5e/include/xrootd/XrdSsi/XrdSsiRespInfo.hh:43:1: warning: 'XrdSsiRespInfo' defined as a struct here but previously declared as a class [-Wmismatched-tags] struct XrdSsiRespInfo ^ /Users/timj/work/lsstsw/stack/DarwinX86/xrootd/u.timj.DM-3584-ge22410fa7f+da39a3ee5e/include/xrootd/XrdSsi/XrdSsiSession.hh:45:1: note: did you mean struct here? class XrdSsiRespInfo; ^~~~~ struct {code} {code} core/modules/xrdoss/QservOss.h:64:17: warning: 'lsst::qserv::xrdoss::FakeOssDf::Opendir' hides overloaded virtual function [-Woverloaded-virtual] virtual int Opendir(const char *) { return XrdOssOK; } ^ /Users/timj/work/lsstsw/stack/DarwinX86/xrootd/u.timj.DM-3584-ge22410fa7f+da39a3ee5e/include/xrootd/XrdOss/XrdOss.hh:63:17: note: hidden overloaded virtual function 'XrdOssDF::Opendir' declared here: different number of parameters (2 vs 1) virtual int Opendir(const char *, XrdOucEnv &) {return -ENOTDIR;} ^ {code} {code} In file included from core/modules/xrdsvc/SsiSession.h:32: /Users/timj/work/lsstsw/stack/DarwinX86/xrootd/u.timj.DM-3584-ge22410fa7f+da39a3ee5e/include/xrootd/XrdSsi/XrdSsiResponder.hh:177:27: warning: control may reach end of non-void function [-Wreturn-type] } ^ {code}",1 DM-3899,"support shared_ptr","I would like to write some functions that return afw::math::Statistics objects and wrap them with SWIG. Unfortunately SWIG requires that any object returned by value must have a default constructor, and Statistics does not. Rather than try to add such an object I propose to make my functions return a shared_ptr to Statistics. This ticket is a request to support that by adding the following to statistics.i: {code} %shared_ptr(lsst::afw::math::Statistics); {code} ",2 DM-3900,"Review of [DM-2983]","I was asked for a revision of [DM-2983] which is part of the Backport HSC parallelization code",4 DM-3901,"Update some tests to support nose and/or py.test","When {{sconsUtils}} is migrated to use {{nose}} or {{py.test}} some test scripts will need to be modified because test discovery will be slightly different and the namespace of test execution will change. Two things to consider: * People would still like the option of running a test as {{python tests/testMe.py}}. * We have to work out how to run the memory test case. ",4 DM-3902,"Fix protobuf compiler warning with clang",NULL,1 DM-3903,"Base Site Data Access Center Description Page",https://confluence.lsstcorp.org/display/~petravick/Data+Access+Center,1 DM-3905,"Gathering use cases for verification data sets","Seeking out developer use cases of incoming data sets. Need to determine if datasets will be accessed for verification only or by developers and QA in general. Determine access methods. ",3 DM-3906,"Future infrastructure, common IT, and facility planning","Includes consulting on developing use cases for Base Site Commissioning Cluster Jason Alt, Paul Wefel, Tom Durbin",13 DM-3907,"Specify FY15 Equipment Purchasing Plan","The hardware contract was finalized at the end of July 2015, leaving 2 months of FY15 for spending the fiscal year hardware budget. In lieu of an annual acquisition strategy document, we draft a one-off FY15 purchasing plan. Assignees: Jason Alt, Bill Glick, Paul Wefel Duration: September 2015",9 DM-3908,"W16 Work on Alert Production Simulator","Continued development of Alert Production simulator. Assignees: Steve Pietrowicz Duration: September 2015 - February 2016",71 DM-3909,"W16 Work on OCS Software Integration","Integrate the OCS Software, delivered from the Camera Team, into AP. Assignees: Steve Pietrowicz Duration: November 2015 - February 2016",38 DM-3910,"Run and document multi-node test with docker","In order to validate Docker setup on CC-IN2P3 cluster, it is required to launch some test on consistent data. S15 LargeScaleTest data doesn't seems to be compliant with latest Qserv version so running multi-node test would be interesting. Nevertheless the multi-node setup doesn't seems to be documented and, hence, is difficult to reproduce.",3 DM-3911,"HSC backport: avoid I/O race conditions config write out","This is a port of [HSC-1106|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1106] When running tasks that write out config settings files ({{processCcd.py}}, for example), if multiple processes start simultaneously, an I/O race condition can occur in writing these files. This is solved here by writing to temp files and then renaming them to the correct destination filename in a single operation. Also, to avoid similar race conditions in the backup file creation (e.g. config.py~1, config.py~2, ...), a {{--no-backup-config}} option (to be used with --clobber-config) is added here to prevent the backup copies being made. The outcome for this option is that the config that are still recorded are for the most recent run.",1 DM-3912,"some ctrl_events tests execute outside of execution domain","There are a couple of ctrl_events tests that attempt to execute outside of the valid domains acceptable by the tests, when they shouldn't be. There's a check in place for tests to find this, but a couple of the tests do not have this check.",1 DM-3913,"transfer and update orchestration documentation","The self-service orchestration documentation needs to be transferred from Trac to Confluence, and updated.",2 DM-3916,"Misc work for this reporting week. ","Deal with comments, coordinate with Jason Alt. Deal with comments on existing work from KT and GDF",2 DM-3917,"Begin thinking about governance aspects of DM operations","Thinking, (but not delivered use case) -- what is the flow of tickets and the division of use cases between the Science and Data Operations? What kind of tooling needs can be assumed to exist to support service management what does that mean for the support of processes? (specifically looked at a summary of the state the market, and took specifics look at what ITSM process supporting tools are open source. -- as may be useful to prototype processes before committing to something.. Looked at several, ITOP seemed to be mature/documented to a level that may be useable. ",4 DM-3918,"Conduct, document, initial follow through on Sept JCC meeting ","Developed agenda items, considered JCC meeting. Minutes are on the LSST confluence. A major item of discussion was related to operations, since we are told this is a priority. Befall followup on CCIN2P3 ISM practices. ",1 DM-3919,"General Management ","Hiring, Internal relationships within the NCSA organization. LSST meetings, general management",4 DM-3920,"Replace boost::regex with std::regex","Boost 1.59 causes a ""keyword hidden by macro"" warn under clang in the regex package. We should be using std::regex now anyway, so this is a good motivator to go ahead and convert.",1 DM-3921,"Create and deploy a git-lfs prototype","Create and deploy a git-lfs-s3 server. High level requirements: * The server should use github to authenticate users. Any github user who is a member of the lsst organization has write access. This means they can push objects to the git-lfs server. * The server should allow for anonymous read access. This means anyone can clone, pull, fetch, etc. * Use an S3 compliant API to store objects. * Simple, well defined method to redeploy. * Uses https. * Backs up to Amazon Glacier (or similar) periodically. The data should be ""slow"" so backing up approximately once a day is OK.",64 DM-3922,"Update multi-node setup documentation","Workers in multi-node setup no longer require granting mysql permissions for test datasets since direct mysql connections are no longer used by the data loader.",1 DM-3923,"W16 ISO Work","Continuous work as LSST ISO. Assignees: Alex Withers Duration: September 2015 - February 2016",39 DM-3924,"Centralize Sphinx configuration for Design Documents","Centralize Sphinx configuration for design documents in {{documenteer}} and provide a facility for design document authors to use YAML files to store document metadata rather than editing {{conf.py}} files.",2 DM-3926,"Implement iostream-style formatting in log package","Implement proposed in RFC-96 change to log macros. This ticket only covers defining new set of macros (LOGS() and friends) which use ostringstream for formatting messages. Migration of all clients and removal of LOGF macros will be done in separate ticket.",1 DM-3928,"port dax_*serv to python3",NULL,3 DM-3929,"Handle queries with no database","Sqlalchemy is generating some queries that are currently killing czar, the list is: {code} set autocommit=0 SHOW VARIABLES LIKE 'sql_mode' SELECT DATABASE() SELECT @@tx_isolation show collation where `Charset` = 'utf8' and `Collation` = 'utf8_bin' SELECT CAST('test plain returns' AS CHAR(60)) AS anon_1 SELECT CAST('test unicode returns' AS CHAR(60)) AS anon_1 SELECT CAST('test collated returns' AS CHAR CHARACTER SET utf8) COLLATE utf8_bin AS anon_1 SELECT 'x' AS some_label select @@version_comment limit 1 {code} Czar should survive unfriendly syntax, and this will be addressed through DM-3764. In this story we will make sure that sqlalchemy-generated queries are properly handled (not just these particular queries, but all queries that do not involve any database and any table). We should run such queries on a local mysql instance (alternatively, perhaps redirect to one of the workers?)",4 DM-3930,"AP, Co-add, Image cache definitions","Added physical breakdown for Alert Postage Stamp Images file system, co-add images file system and image cache file system",2 DM-3931,"sandbox-stackbuild issues","A number of issues with sandbox-stackbuild, or rather its infrastructure. 1. Problem with the librarian-puppet plugin and its mismatch with the puppet forge API ([~jhoblitt] has a PR open apparently) 2. As a workaround to above, one needs to {code} gem install librarian-puppet librarian-puppet install {code} but that runs into an issue with swap_file needing a downgrade to work with Ubuntu 14.04. Working state as of the time of this bug report for the Puppetfile is: {code} forge 'https://forgeapi.puppetlabs.com' mod 'puppetlabs/stdlib' mod 'camptocamp/augeas', '~> 1.4' mod 'stahnma/epel', '~> 1.1' mod 'petems/swap_file', '1.0.1' mod 'jhoblitt/sysstat', '~> 1.1' mod 'maestrodev/wget', '~> 1.7' mod 'jhoblitt/lsststack', :git => 'https://github.com/lsst-sqre/puppet-lsststack.git' {code} 3. Which brings us to the fact that the Vagrant puppet-install plugin is broken with Puppet 4, and new platforms are not supported under Puppet 3. Ergo, as is, can't bring up Ubuntu 15.05 etc. Ticket is to get PRs merged, fork and fix them ourselves, or find alternatives. ",4 DM-3932,"Histogram calculation for image stretch has infinite loop ","When load the big.fits file, the image never came out. It stopped at Histogram. There was an infinity in Histogram. ",4 DM-3935,"Measurement plugin errors","When doing measurements on coadds, several errors are thrown within the measurement plugins. {code} Error in base_GaussianFlux.measure on record 283467884979: Input shape is singular {code} {code} Error in base_GaussianFlux.measure on record 283467883979: File ""src/SdssShape.cc"", line 842, in static lsst::meas::base::FluxResult lsst::meas::base::SdssShapeAlgorithm::computeFixedMomentsFlux(const ImageT&, const lsst::afw::geom::ellipses::Quadrupole&, const Point2D&) [with ImageT = lsst::afw::image::MaskedImage; lsst::afw::geom::Point2D = lsst::afw::geom::Point] Error from calcmom {0} lsst::pex::exceptions::RuntimeError: 'Error from calcmom' {code} The measurements were done on tiger, and the command used was: {code} measureCoaddSources.py /tigress/HSC/HSC/rerun/nate/old_clip/ --output=/tigress/HSC/HSC/rerun/nate/old_clip/ -C /home/nlust/options_temp.py --id tract=0 patch=2,2 filter=HSC-I^HSC-R {code} The data can be found within the rerun directory specified as the input to the command. The data was created using the commands: {code} assembleCoadd.py --legacyCoadd /tigress/HSC/HSC/rerun/nate/ --output=/tigress/HSC/HSC/rerun/nate/old_clip --id tract=0 patch=2,2 filter=HSC-R --selectId visit=1208 ccd=56^64^72 --selectId visit=1206 ccd=64^65^72^73^79^80 --selectId visit=1212 ccd=64^65^72^73^79^80 --selectId visit=23704 ccd=64^65^72^73 --selectId visit=23706 ccd=56^64^72 --selectId visit=23694 ccd=64^65^72^73^79^80 --selectId visit=1204 ccd=64^65^72^73^79^80 --selectId visit=1220 ccd=63^64^71^72^78^79 --selectId visit=1218 ccd=56^57^64^65^72^73 --selectId visit=23718 ccd=64^65^72^73^79^80 --selectId visit=23692 ccd=64^65^72^73^79^80 --selectId visit=1210 ccd=63^64^71^72^78^79 --selectId visit=1216 ccd=56^57^64^65^72^73 --selectId visit=1214 ccd=64^65^72^73^79^80 --selectId visit=23716 ccd=63^64^71^72^78^79 --selectId visit=1202 ccd=64^65^72^73^79^80 {code} and {code} assembleCoadd.py --legacyCoadd /tigress/HSC/HSC/rerun/nate/ --output=/tigress/HSC/HSC/rerun/nate/old_clip --id tract=0 patch=2,2 filter=HSC-I --selectId visit=19658 ccd=64^65^72^73^79^80 --selectId visit=1248 ccd=56^64^72 --selectId visit=19696 ccd=65^66^73^74^80^81 --selectId visit=19684 ccd=64^65^72^73^79^80 --selectId visit=1238 ccd=64^65^72^73^79^80 --selectId visit=19710 ccd=56^64^72 --selectId visit=19680 ccd=56^64^72 --selectId visit=1230 ccd=64^65^72^73^79^80 --selectId visit=1236 ccd=63^64^71^72^78^79 --selectId visit=19694 ccd=64^65^72^73^79^80 --selectId visit=1232 ccd=64^65^72^73^79^80 --selectId visit=19698 ccd=64^65^72^73^79^80 --selectId visit=1228 ccd=64^65^72^73^79^80 --selectId visit=1246 ccd=63^64^71^72^78^79 --selectId visit=19682 ccd=63^64^71^72^78^79 --selectId visit=19708 ccd=64^65^72^73^79^80 --selectId visit=19662 ccd=64^65^72^73 --selectId visit=1240 ccd=64^65^72^73^79^80 --selectId visit=1244 ccd=56^57^64^65^72^73 --selectId visit=1242 ccd=56^57^64^65^72^73 --selectId visit=19660 ccd=64^65^72^73^79^80 --selectId visit=19712 ccd=56^57^64^65^72^73 {code}",6 DM-3936,"Fix ""Executive error executing job"" on the cluster",NULL,1 DM-3937,"HSC backport: temporary file handling in butler","The HSC fork includes additional work to improve temporary file usage in the butler: * [HSC-1275|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1275]: Probable resource leakage by butler * [HSC-1285|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1285]: eups.version files ignore umask * [HSC-1292|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1292]: Prevent opening files that are already open",1 DM-3938,"Pastry prototype C++",NULL,22 DM-3939,"move camera factory methods from obs_lsstSim to afw","The methods defined in obs_lsstSim/bin/makeLsstCameraRepository.py can be easily adapted for use in generating arbitrary, non-LSST cameras. This is useful for the sims stack, both for testing purposes, and because members of other projects have begun asking us to use our code. This ticket will take those methods, make them fully LSST-agnostic, and place them in afw as utility functions. The code in obs_lsstSim will refer back to these afw methods.",5 DM-3940,"NaiveDipoleCentroid/NaiveDipoleFlux algorithms should not require centroid slot","The {{NaiveDipoleCentroid}} and {{NaiveDipoleFlux}} algorithms in {{ip_diffim}} have members which are instances of {{meas::base::SafeCentroidExtractor}}. Due to the prerequisites that imposes, it is impossible to initialize these algorithms without first defining a {{centroid}} slot. However, there is nothing in these algorithms which actually uses the {{SafeCentroidExtractor}} or any of the information stored in the slot; this seems to be an entirely arbitrary restriction which is likely a legacy of the port to the {{meas_base}} framework. We should remove the use of {{SafeCentroidExtractor}} to simply the code and make it easier to run the test suite (since it will no longer be necessary to run a centroider).",1 DM-3941,"Split FY16 plan into audience specific documents","Fy16 purchase planning needs to be split for specific audiences (NCSA planning, Aura purchase approval)",2 DM-3942,"Preliminaries for the LSST vs. HSC pipeline comparison through single-frame processing","This ticket is in preparation for DM-2984, which is to run both the HSC and LSST pipelines on 2-3 visits of HSC data, and do a detailed comparison of the science quality and robustness for the single-frame processing (ProcessCcdTask) stage only. Essentially, it is a detailed audit of all the functionality currently used in HSC single-frame processing and ensuring all relevant features have been pulled over to the LSST stack such that the comparisons to be done in DM-2984 will be meaningful and informative.",14 DM-3943,"QMeta thread safety","Initial QMeta implementation is not thread safe, it uses sql/mysql modules which also do not have any protection (there are some mutexes there but not used). Need an urgent fix to avoid crashes due to concurrent queries in czar.",1 DM-3944,"orchestration slide set for DM bootcamp","Create the ""Orchestration and Control"" slide set for DM Bootcamp, which is being held from Oct 5-7, 2015. After review (and any revisions), the slide set will be uploaded to confluence, and a link to it will be put here.",5 DM-3945,"Simplify task queuing / Runner code",NULL,8 DM-3946,"Cleanup cancellation-related worker code",NULL,7 DM-3947,"Remove dependency on mysqldb in wmgr","Move remaining code that depends on mysqldb to db module",1 DM-3949,"Remove dependency on mysqldb in qserv","Remove remaining dependencies on mysqldb in qserv.: {code} ./core/modules/tests/MySqlUdf.py ./core/modules/wmgr/python/config.py {code} and use the sqlalchemy from db module instead.",2 DM-3951,"Remove qserv_objectId restrictor","qserv_objectId restrictor can be replaced by the IN restrictor. This story involves checking if performance is acceptable if we use IN restrictor instead of qserv_objectId restictor, and if it is, doing the switch and removing the qserv_objectId restictor code.",3 DM-3952,"Cleanup lua miniParser","Maybe some cleanup can also be performed in lua code. Indeed ""objectId"" hint and parseObjectId() which seems useless. Indeed miniParser.parseIt and miniParser.setAndNeeded seems useless. Removing this code will ease maintenance of objectId management.",1 DM-3954,"E/I training and interview","Interviewed and attended training for E/I concerns.",1 DM-3955,"Investigate services for backups/data replication on Nebula openstack","Files generated on instances of the Nebula openstack should be managed with some commensurate level of data replication/backups. We investigate services that might serve this task within the cloud context.",4 DM-3956,"Package SQLAlchemy in eups","Db module is expected to be used by science pipelines, and (per K-T, see qserv hipchat room) we have to package it through eups.",1 DM-3957,"Enable CModel in CalibrateTask prior to PhotoCal","CModel needs to run in CalibrateTask before PhotoCal in order to compute aperture corrections, but it also needs a Calib objects as input, and that isn't available until after PhotoCal is run. On the HSC side, we dealt with this by adding preliminary PhotoCal run before CModel is run, but we could also deal with it by removing the need for a Calib as input, at least in some situations.",1 DM-3958,"Revisit provenance design / built proof-of-concept prototype","Discuss existing provenance design with the team, brainstorm and improve. Primary focus on capturing provenance information. Deliverable: proof-of-concept prototype + description.",16 DM-3959,"Revisit provenance sizing","Revisit estimates of the size of provenance information.",6 DM-3960,"First optimizations of provenance querying","Think through the issues of querying provenance. Deliverable: write up attached to this story. IT will be transferred to more official provenance documentation through DM-3961",14 DM-3961,"Document provenance design",NULL,8 DM-3962,"Build prototype of provenance","Building a proof-of-concept prototype of provenance. Deliverable: a working, standalone (not connected to any data producer) prototype of the Data Provenance. Not optimized / alpha version.",12 DM-3965,"Meetings Sep 2015","verification datasets weekly meetings and tech-talks",1 DM-3966,"Add SQLite-based v01. unit tests for dbserv",NULL,6 DM-3967,"Meetings, institute events, or other LOE, Sep 2015","NCSA or Astronomy Department activities. - NCSA All-hands - Local LSST group meetings - DES-Illinois meetings - Colloquia - other seminars, info sessions, or other local meetings",5 DM-3968,"detailed out alert processing/ transmission to event brokers. ","Detailed out the interactions with the event broker, after consulting with John Swinbank. Dealt with outputs of alert processing. Dealt with corrections and comments. Began considering annual processing",2 DM-3969,"weekly management","Dealt with impending new hire. Good deal of intergroup coordination, and deal with NCSA re organization.",13 DM-3971,"Package sqlalchemy in eups","Db module is expected to be used by science pipelines, and (per K-T, see qserv hipchat room) we have to package it through eups.",1 DM-3973,"FY pricing estimates",NULL,1 DM-3974,"Network / Storage reviews","Initial (internal) review with networking and storage delegates",2 DM-3975,"research items needed for wan simulator procurement","locate part numbers and pricing information",1 DM-3976,"Consultation on system and network design",NULL,4 DM-3977,"Prepare pricing estimates for networking infrastructure","provide switch quantities, weight, power and budgetary costs",4 DM-3978,"Recommend network infrastructure for Openstack expansion",NULL,1 DM-3979,"Discussion of Base site network proposal","Email based discussion with Ron Lambert on his network infrastructure proposal for the Base site.",1 DM-3980,"Post SQLAlchemy-migration tweaks","Implement some minor tweaks take came in late through PR comments, mostly related to sqlalchemy related migration",1 DM-3981,"Improve the performance for making the image plot","Make FitsRead work better for multi-threads",1 DM-3983,"Larger Statistics needed for CModel Studies","The stampsize and nInitalRadius tests were not conclusive in the September Sprint. and the error estimates appeared to be overly large. The nGrowFootprints test was barely significant. This is a continuation of work started on DM-1135 (DM-1135, 3375, 3376) , after a study of the sizes of our error estimates was conducted (DM-3984). We started with a sample with 12 million galaxies (not all of which were used in DM-3375 and 3376. They appeared to all be useful once we had new error estimates, so the studies were run against with this larger sample. ",6 DM-3984,"Errors for shear bias fits","DM-1135 mostly was inconclusive or at least not highly significant, due to the fact that the error bars were around the same size as the differences in most of the tests. This in spite of the fact that we ran 6x as many sample galaxies as great3sims. Investigate the reason for this, and see if we can estimate the errors more accurately. Investigate what the best way (or at least the way it is done in GREAT3) to estimate the errors on the bias parameters. If it is not from the covariance matrix of the regression parameters, there could be some work here.",6 DM-3985,"Run multinode test using docker on one unique host","Qserv multinode test can be launched on n hosts using docker, but not on one unique host. This ticket will allow developers to run multinode test on their workstation.",6 DM-3986,"Deploy developer code on in2p3 cluster in Docker images","Qserv latest release can be deployed easily on cc-in2p3 cluster using Docker. This ticket will allow developers to prepare worker and master containers using a specific Qserv version and deploy it on cc-in2p3 cluster.",6 DM-3987,"remove unnecessary 'psf' arg to SourceDeblendTask.run()","{{SourceDeblendTask.run}} takes both an {{Exposure}} and a {{Psf}}, even though it can get the latter from the former and always should.",1 DM-3988,"Update FY2015 hardware budget plan","Update the FY2015 hardware purchasing plan with new budget, equipment specifications, and general costs. ",3 DM-3989,"Week end 09/05/15","Support for lsst-dev cluster, OpenStack, and accounts for week ending September 5, 2015.",5 DM-3990,"Provide detail specs to AURA","Provided Josh Hobblitt details specs for procurement request",1 DM-3992,"Week end 09/12/15","Support for lsst-dev cluster, OpenStack, and accounts for week ending September 12, 2015.",5 DM-3993,"Display.dot origin swaps x and y","Correcting for xy0 in {{dot}} currently does: {code:hide-linenum} r -= x0 c -= y0 {code} which is backwards.",1 DM-3994,"Backup Pugsley","Created backup of pugsley in anticipation of new hardware and shutdown of temp Mac OS solution",1 DM-3996,"Research using vSphere on Mac Pro","Researched setting up vSphere on Mac Pro.",1 DM-3997,"Week end 09/19/15","Support for lsst-dev cluster, OpenStack, and accounts for week ending September 19, 2015.",5 DM-3998,"Learn about Lenovo storage and server options","Went to lunch with Lenovo to learn about systems that might be suitable for LSST deployment. Learned about server and storage options.",1 DM-3999,"Week end 09/26/15","Support for lsst-dev cluster, OpenStack, and accounts for week ending September 26, 2015.",5 DM-4000,"Revamp FITS Viewer scrolling to stop using large div","The current scrolling system will not work as well with masking layer. It is also suspected to use too much browser memory since is creates a very large div. Firefly will scroll images manually now.",12 DM-4001,"Write next-generation stack doc writing guide","Write a guide in the prototype LSST Stack Docs (https://github.com/lsst-sqre/lsst_stack_docs) covering how to document the LSST Stack under the new doc infrastructure. This exercise will implicitly involve designing how the new docs will work. Content includes: # How to write a user guide to a package (both content wise and in terms of organizing a package's doc files) # How to write python doc strings # Coverage of reStructuredText and Sphinx as implemented by the LSST Stack Docs. ",7 DM-4003,"Replace zookeeper CSS with mysql","To switch from QservAdmin to CssAccess interface in our Python tools we will need to replace zookeeper with mysql implementation because we do not have C++ KvInterface implementation for zookeeper.",2 DM-4004,"Investigate what is missing to run ISR with DECam raw data and processCcd","Currently raw DECam data can not be processed with the stack for Instrumental Signature Removal (ISR). This issue includes efforts to understand how ISR is done in the stack, learn about processCcd, basic DECam ISR, and what is missing in the code stack to proceed. ",13 DM-4007,"ctrl_execute templates still use ""root"" instead of ""config""","The templates that ctrl_execute fills in still use ""root"", instead of the new ""config"". This causes extraneous warning messages to appear from pex_config when executing ""runOrca.py""",1 DM-4009,"Allow FlagHandler to be used from Python","The {{FlagHandler}} utility class makes it easier to manage the flags for a measurement algorithm, and using it also makes it possible to use the {{SafeCentroidExtractor}} and {{SafeShapeExtractor}} classes. Unfortunately, its constructor requires arguments that can only be provided in C++. A little extra Swig wrapper code should make it usable in Python as well.",3 DM-4011,"Rename forced photometry CmdLineTasks to match bin scripts","We use names like ""forcedPhotCcd.py"" for bin scripts but ""ProcessCcdForcedTask"" for class names; these need to be made consistent, and it's the former convention that was selected in an old (non-JIRA) RFC.",1 DM-4013,"Initial discussion EFD with Dave Mills","Review LTS-210 and discuss design of EFD cluster with Dave Mills. Are there opensource clustering solutions?",1 DM-4014,"Replace boost::tuple with ","Replace boost::tuple with This ticket will be completed as part of the DM bootcamp at UW.",1 DM-4015,"Document Revision","Document cleanup. Added PDUs, networking estimates, power costs, vsphere licensing.",1 DM-4016,"Write developer workflow documentation","Write a developer workflow guide to walk through and document best practices for developing against the LSST Stack.",6 DM-4019,"Fix procedure for building docker image for 2010_09 release","This procedure should be straightforward but is currently failing due to gcc-4.9/boost problem (DM-4018)",2 DM-4021,"Replace boost::unordered_map with std::unordered_map","DM boot camp tutorial. ",2 DM-4022,"forcedPhotCoadd.py fails on HSC data","When trying to run {{forcedPhotCoadd.py}} on HSC data, I see the following error: {code} $ forcedPhotCoadd.py /raid/swinbank/rerun/LSST/bootcamp --id filter='HSC-I' tract=0 patch=7,7 : Loading config overrride file '/nfs/home/swinbank/obs_subaru/config/forcedPhotCoadd.py' Cannot import lsst.meas.extensions.photometryKron: disabling Kron measurements Traceback (most recent call last): File ""/home/lsstsw/stack/Linux64/meas_base/11.0+2/bin/forcedPhotCoadd.py"", line 24, in ForcedPhotCoaddTask.parseAndRun() File ""/home/lsstsw/stack/Linux64/pipe_base/11.0+2/python/lsst/pipe/base/cmdLineTask.py"", line 433, in parseAndRun parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides) File ""/home/lsstsw/stack/Linux64/pipe_base/11.0+2/python/lsst/pipe/base/argumentParser.py"", line 360, in parse_args self._applyInitialOverrides(namespace) File ""/home/lsstsw/stack/Linux64/pipe_base/11.0+2/python/lsst/pipe/base/argumentParser.py"", line 475, in _applyInitialOverrides namespace.config.load(filePath) File ""/home/lsstsw/stack/Linux64/pex_config/11.0/python/lsst/pex/config/config.py"", line 529, in load self.loadFromStream(stream=code, root=root) File ""/home/lsstsw/stack/Linux64/pex_config/11.0/python/lsst/pex/config/config.py"", line 549, in loadFromStream exec stream in {}, local File ""/nfs/home/swinbank/obs_subaru/config/forcedPhotCoadd.py"", line 10, in config.deblend.load(os.path.join(os.environ[""OBS_SUBARU_DIR""], ""config"", ""deblend.py"")) AttributeError: 'ForcedPhotCoaddConfig' object has no attribute 'deblend' {code} This is with the stack version 11.0+3 and {{obs_subaru}} 5.0.0.1-676-g4ae362c.",1 DM-4023,"Local LSST IAM Meeting","Meeting notes: * [10 minutes] Review LSST identity management statement of work On the confluence wiki. Let me know if you have any follow-up questions/comments. * [10 minutes] Plan initial project tasks All: Review materials at https://confluence.lsstcorp.org/display/LAAIM. Jim to add updates from last week's meeting. * [5 minutes] Schedule follow-on project meetings Alex will schedule meeting with Bill Glick about LDAP at NCSA. Terry and Daniel will attend https://community.lsst.org/t/dm-boot-camp-announcement/249 sessions of interest if available.",1 DM-4024,"Local LSST IAM Meeting","-Meeting attended by Jim Basney, Tim Fleury, Dan Thayer, and Alex Withers. -Discussed upcoming DM bootcamp. -Focused on SUIT diagram and determining what questions to ask at next bi-weekly meeting. -First draft recommendations by end of October.",1 DM-4025,"First day activities","Meeting with HR at 8:30AM. Taken around for introductions through 10AM. Remainder of day was proceeding through HR punch list for new hires such as: NCSA Intranet account setup Gain familiarity with NCSA Intranet Kerberos and Enterprise ID setup Read and Sign-off on Security and NCSA policy. Outlook mail and calendar setup 2pm DM meeting",2 DM-4026,"Second day start activities","Requisitioned used macbook and monitor from IT and set it up. Located LSST stack site and read build and install documentation More HR account items. Requested confluence credentials from LSST.org Learned about SDSS/Stripe 82",2 DM-4027,"groundwork for file management","Installed correct Xcode + misc. for stack version. Downloaded LSST stack from GitHub repo and built/installed. ",1 DM-4028,"Incidental items and jobs for file mgmt. groundwork","Installed and built LSST tutorials package. Setup, fixed minor issues and ran First tutorial to check that initial stack installation was successful. Learned about CCD operation and how the LSST CCD is laid out. Learned about raw CCD data from amplifiers, as well as other camera attributes Studied stack code for these things. Read description of Astronomy Associative relations as well as CoAdds. Pulled PhoSim from repo. Built and Installed. Had trouble getting PhoSim to run due to dynamic link library issues. Instrumented PhoSim code with PDB commands. Walked to bookstore and picked up new ID badge.",2 DM-4029,"Development for Developer activities!","Completed setup of confluence access. Set up LSST HipChat. Jumped into PhoSim a bit more and resolved errors by dropping in a few symlinks here and there; not a solution for a proper execution environment, but a time-save just to see PhoSim work and have a platform for tinkering with camera attar's. PhoSim ran successfully. Learned about FTTS format and how the file is built up. Learned about how Key/Val pairs can proceed ANY data section throughout file by using data offset values.",2 DM-4030,"integration of confluence data into learning curve","Started studying LSST coding policies and best practices via confluence. Colloquium. Small meetings with other LSST team members throughout day. Rebuilt PhoSim/LSST-Stack to take advantage of multiple cores when rendering portion of sky to a file. Built these commands into Stack code. They would have to be custom #defined by ./configure at build time depending on computer arch. which is too much to do when just gaining familiarity so stuck to MacBook multi-core specs, where I was working. Logged into JIRA and studied how tasks were proposed, realized, and checked as done. Extensive talk about BBQ in group area of NCSA/LSST. Close reading and note taking of LDM-230 and related docs.",5 DM-4031,"Add doc directory, and fix doxygen warnings","Add doc directory and fix doxygen warnings.",4 DM-4032,"obs_sdss should use pydl.yanny instead of it's own copy thereof","Inside obs_sdss there is a yanny.py that looks like it was copied from either sdss_python_module or pydl. We should just depend on pydl (https://github.com/weaverba137/pydl), so we can use whatever improvements it gets for free, and to prevent yet annother yanny reader floating around.",4 DM-4034,"Display DECam focal plane mosaics using showCamera","Try out some new functionalities of afw.cameraGeom.utils (from DM-2437) for DECam raw data. Raw data in {{testdata_decam}} are retrieved through Butler and displayed as a focal plane mosaic. ",2 DM-4035,"Replace boost::array with std::array","Replace all use of boost::array with std::array in the DM stack A quick search turned up use in 17 files spread over these packages: ip_diffim, meas_base, meas_extensions_photometryKron and ndarray (which is presumably out of scope for this ticket)",4 DM-4036,"Change from boost::math","Most boost::math contents (not including pi) are now available in standard C++. Please convert the code accordingly. In addition to the packages listed above, boost/math is used in ""partition"" a package I don't recognize and not a component JIRA accepts.",5 DM-4039,"New YAML config for community_mailbot","Per review comments to DM-3690, the configuration should move to YAML with the following goals - Have a Configuration object that can be tested and passed around - Redesign the configuration to allow for additional types of message handlers, such as twitter, hipchat/slack, etc. - Move secret keys entirely into the configuration file - Provide a configuration template - Move any sort of hard-coded configuration to the expanded YAML file (e.g., Mandrill templates)",1 DM-4040,"Refactor Scripts and Discourse interface in community_mailbot","Per review comments for DM-3690, the community_mailbot can have slight code refactoring - Refactor scripts into smaller testable units - Refactor the Discourse feed classes around an ABC - More testing",1 DM-4042,"Produce demo video for git lfs","Produce a screencast tutorial of the DM git-lfs implementation.",1 DM-4043,"update memory management in jointcal","jointcal currently uses a combination of raw pointers and a custom reference-counted smart pointer class, {{CountedRef}} (similar to {{boost::intrusive_ptr}}). The code needs to be modified to use a combination of {{shared_ptr}} (most code), {{unique_ptr}} local-scope variables and factory functions, and {{weak_ptr}} (at least some will be necessary to avoid cycles in some of the more complex data structures). As part of this work, we'll also have to remove a lot of inheritance from {{RefCounted}}, which is part of the {{CountedRef}} implementation. This ticket looks like it will require a lot of work, because we'll have to be careful about every conversion to avoid cycles and memory leaks. Nevertheless, I think it will be necessary to do this conversion before attempting any other major refactoring, as I'm worried that having a newcomer make changes to the codebase without first making the memory management less fragile could be very dangerous.",8 DM-4044,"integrate jointcal geometry primitives (Point, Frame, FatPoint) with afw","jointcal currently has three simple geometry classes that can be integrated relatively easily with existing classes in afw and meas_base: - {{jointcal.Point}} is equivalent to {{afw.geom.Point2D}}. - {{jointcal.Frame}} is equivalent to {{afw.geom.Box2D}}. - {{jointcal.FatPoint}} is equivalent to {{meas.base.CentroidResult}}. We should probably move {{CentroidResult}} to {{afw.geom}}, perhaps rename it ({{MeasuredPoint}}?), and reconsider its relationship with {{Point}}. This will require a bit of refactoring in {{meas_base}}, but the usage in {{jointcal}} makes me think it's a sufficiently fundamental object to be included in afw. We may find aspects of the interfaces in {{jointcal}} that we should add to {{afw.geom}}, but I think we'll mostly end up making trivial modifications to {{jointcal}} to use the {{afw}} interfaces.",8 DM-4045,"integrate Gtransfo functionality with XYTransform","{{meas_simastrom}} includes a {{Gtransfo}} class hierarchy that is similar to {{afw.geom.XYTransform}}, but with more functionality and some intentional differences, including: - {{XYTransform}} objects are immutable; {{Gtransfo}} objects are not. - {{Gtransfo}} objects expose their parametrization, and can compute various derivatives with respect to those parameters. {{XYTransforms}} are essentially black-box functions, and expose no parameterization. Unifying these classes is not entirely straightforward, and should include an RFC for the design prior to implementation. Overall, I think {{XYTransform}}'s simpler, lower-functionality interface and immutability is worth mostly preserving somehow; I think it's a more fundamental interface than {{Gtransfo}} that can be used in more places. But obviously we need to provide the more extensive {{Gtransfo}} interface somehow as well. My initial thought is that we should have two parallel class hierarchies (with a concrete class for each type of transform, such as polynomial distortion, in both), and an ultimate base class shared by both hierarchies. That ultimate base class would contain most of the current {{XYTransform}} interface but not require immutability, and one side of the tree would contain simple immutable objects while the other would contain the more extensive parameterized interface of {{Gtransfo}}.",8 DM-4046,"Redis cache for community_mailbot","Switch from a {{json}} cache to a redis cache from the community_mailbot.",1 DM-4047,"Add fake secondary index to testIndexMap.cc","qproc/testIndexMap.cc is sketchy and perform a very poor validation for now. It should at least use a minimal SQL secondary index, embedded in a simple database like SQLLite, inorder to validate secondary index lookup code.",8 DM-4048,"Replace QsRestrictor::PtrVector With std::vector and use move constructor","Use of QsRestrictor::PtrVector introduces a useless indirection. it maybe could be replace by std::vector and use of move constructor. This would simplify code (currently a confusion exists between empty vector and nullptr) and ease maintenance.",5 DM-4049,"Security plan renewal",NULL,3 DM-4050,"IaM work",NULL,2 DM-4052,"bi-weekly IaM meeting",NULL,1 DM-4054,"Meetings Oct 2015","- Verification datasets meetings ",1 DM-4055,"Final additions and review","Add remaining components: power estimates, vSphere annual licensing, networking, PDUs, login nodes review: misc expense fund, decommissioned services, financial targets",3 DM-4056,"Chasing down Pan Starrs Requirements","Pan Starrs data release (PS1) will be used in the integration QServ environment purchased as part of FY16. Catalog and file space requirements must be understood.",1 DM-4057,"Prepare Plan for ICI Leadership Review","Prepare plan and ICI-group-specific points of interest for hardware/service deployment plans. ",1 DM-4058,"Vendor Discussions","Discussions with vendors on planned procurement. Details of discussions will not be described here. This is for story point tracking only.",1 DM-4059,"Begin Approval Process","Approval process for FY16 procurement plan. This requires approval from Jeff K, Victor and NSF (due to the cost increment being greater than $250K). Expected approval time frame: Dec 2015.",1 DM-4061,"Define policy based upon FY16 plans","Re evaluate previously proposed storage policies: https://wiki.ncsa.illinois.edu/display/LSST/Storage+Policy?src=contextnavpagetreemode Plan new policies: https://wiki.ncsa.illinois.edu/display/LSST/Changes+to+Storage+Policy+and+Design?src=contextnavpagetreemode",1 DM-4062,"Data access rights and retention policies","Added Data access center requirements, individual data access rights, data retention policies and other general cleanup.",1 DM-4063,"Support new casting requirements in NumPy 1.10","The function imagesDiffer() in testUtils attempts to OR an array of unit16s (LHS) against an array of bools(RHS) {{valSkipMaskArr |= skipMaskArr}} and errors with message {code} TypeError: ufunc 'bitwise_or' output (typecode 'H') could not be coerced to provided output parameter (typecode '?') according to the casting rule ''same_kind'' {code} preventing afw from building correctly. ",1 DM-4064,"Revisit database compression trade-offs","As discussed at Qserv meeting Oct 7, it is not entire clear if it will be worth compressing data. Need to revisit baseline.",5 DM-4065,"Discuss with MySQL team","This story captures issues/topics that we want to bring up with mysql team.",2 DM-4067,"SuperTask structure implementation ","Starting to implement the structure of the Super Task framework for process execution",7 DM-4068,"SuperTask framework extension","Incorporate Configuration and data reference to implementation",4 DM-4069,"Bootcamp meeting","I've attended LSST DM bootcamp",3 DM-4070,"SuperTask framework documentation and refactorization","While still prototyping I need to fill documentation on new code as well as do some clean up as well",7 DM-4071,"testPsfDetermination broken due to NumPy behaviour change","Old NumPy behaviour (tested on 1.6.2): {code} In [1]: import numpy In [2]: a = numpy.array([]) In [3]: numpy.median(a) /usr/lib64/python2.6/site-packages/numpy/core/fromnumeric.py:2374: RuntimeWarning: invalid value encountered in double_scalars return mean(axis, dtype, out) Out[3]: nan {code} New NumPy behaviour (1.10.0): {code} In [1]: import numpy In [2]: a = numpy.array([]) In [3]: numpy.median(a) [...] IndexError: index -1 is out of bounds for axis 0 with size 0 {code} This breaks {{testPsfDeterminer}} and {{testPsfDeterminerSubimage}}, e.g.: {code} ERROR: testPsfDeterminerSubimage (__main__.SpatialModelPsfTestCase) Test the (PCA) psfDeterminer on subImages ---------------------------------------------------------------------- Traceback (most recent call last): File ""./testPsfDetermination.py"", line 342, in testPsfDeterminerSubimage trimCatalogToImage(subExp, self.catalog)) File ""/Users/jds/Projects/Astronomy/LSST/src/meas_algorithms/python/lsst/meas/algorithms/objectSizeStarSelector.py"", line 377, in selectStars widthStdAllowed=self._widthStdAllowed) File ""/Users/jds/Projects/Astronomy/LSST/src/meas_algorithms/python/lsst/meas/algorithms/objectSizeStarSelector.py"", line 195, in _kcenters centers[i] = func(yvec[clusterId == i]) File ""/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/lib/function_base.py"", line 3084, in median overwrite_input=overwrite_input) File ""/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/lib/function_base.py"", line 2997, in _ureduce r = func(a, **kwargs) File ""/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/lib/function_base.py"", line 3138, in _median n = np.isnan(part[..., -1]) IndexError: index -1 is out of bounds for axis 0 with size 0 {code}",1 DM-4074,"S18 Improve Webserv",NULL,79 DM-4075,"Assemble eslint rules for JavaScript code quality control","Review and assemble eslint rules, which enforce clean JavaScript and JSX code. Code cleanup to avoid too many rule violations.",8 DM-4076,"JavaScript code cleanup - remove unused packages","Remove es6-promise, react-modal, other cleanup",2 DM-4078,"convert underscore to lodash","lodash has become a superset of underscore, providing more consistent API behavior, more features, and more thorough documentation. We'd like to convert our underscore package dependencies to lodash while we have only ~20 calls to underscore functions",2 DM-4079,"Create a React component which manages tabs","We need a React component, which manages tabs. ",6 DM-4080,"Shutdown mechanism doesn't work when logging process is disabled.","If the logging mechanism is turned off in ctrl_execute, the ctrl_orca Logger doesn't get launched. The current shutdown mechanism waits for the last logging message to be transmitted before shutting down so it doesn't kill off that process. If the logger.launch config file option is set to false, this process never get launched and ctrl_orca hangs after the shutdown waiting for the message to arrive.",8 DM-4081,"Lead the Firefly conversion from GWT to React/FLUX design meeting","A focused week long design meeting on Firefly conversion from GWT to React/FLUX. Agenda and notes here https://confluence.lsstcorp.org/pages/viewpage.action?pageId=41786446 Design document at https://confluence.lsstcorp.org/display/DM/Firefly+Client+Application+Architecture ",6 DM-4082,"Firefly conversion from GWT to React/FLUX design meeting","A focused week long design meeting on Firefly conversion from GWT to React/FLUX. Produce the first draft of design document https://confluence.lsstcorp.org/display/DM/Firefly+Client+Application+Architecture",10 DM-4083,"Firefly conversion from GWT to React/FLUX design meeting","A focused week long design meeting on Firefly conversion from GWT to React/FLUX. Design document at https://confluence.lsstcorp.org/display/DM/Firefly+Client+Application+Architecture ",6 DM-4084,"Firefly conversion from GWT to React/FLUX design meeting","A focused week long design meeting on Firefly conversion from GWT to React/FLUX. Design document at https://confluence.lsstcorp.org/display/DM/Firefly+Client+Application+Architecture. ",4 DM-4085,"Attend DM boot camp","Attend DM boot camp to learn more about DM stack, butler, and task. ",2 DM-4086,"Attend DM boot camp","Attend DM boot camp to learn more about DM stack, butler, and task. Most of the presentations are located at URL https://community.lsst.org/t/dm-boot-camp-announcement/249. Presentations like afw, eups, tasks, and butler are necessary to participate in LSST, so everyone on LSST must understand these concepts. Look at the list of presentations covering these topics and make sure your understand them. Some of the remaining talks go into more detail or cover more specialized topics. Those talks should be scanned to see if they are of interestß to you.",3 DM-4087,"Attend DM boot camp ","Attend DM boot camp to learn more about DM stack, butler, and task. ",3 DM-4089,"Consulting in September",NULL,3 DM-4090,"update cat logging information","The ""cat"" package has a table which the Logger in ctrl_orca uses to insert information from logging messages. The format of the log messages has changed, and therefore the table in ""cat"" needs to be changed as well.",1 DM-4092,"Update qserv for lastest xrootd","Small API change in latest xrootd, requires a parallel change to qserv. Paves the way for DM-2334",1 DM-4093,"Update DECam camera geometry descriptions for raw data","The overscan and prescan regions of instcal data have been trimmed, but they are included for raw data. The amplifier information in the current camera descriptions were made for instcal data and do not include overscan and prescan regions. While processing raw data with the current camera descriptions, the bounding boxes from the camera object seem incorrect for raw data. Code change summary: - Use non-zero overscan and prescan regions - Update the pixel array layout. My schematic of pixel array layout is attached in DecamAmpInfo.png Screenshots are post-ISR images processed with bias and flat correction using the old or updated camGeom. ",7 DM-4094,"Educational Activities for In-Depth Reusable Background","Bucket epic to capture effort spent in educational activities and meetings to gain in-depth reusable background knowledge for the LSST project.",10 DM-4095,"Please port showVisitSkyMap.py from HSC","The HSC documentation at http://hsca.ipmu.jp/public/scripts/showVisitSkyMap.html includes a useful script for displaying the skymap and CCDs from a set of visits. It would be convenient if a version of this script was available in the LSST stack.",1 DM-4098,"Update Trust Level of all LSST DM Staff to Level 4 via the API","It seems safe to update the Discourse trust level of all members of the LSSTDM group on community.lsst.org to Level 4 (full permissions). See https://meta.discourse.org/t/consequences-of-using-or-bypassing-trust-levels-for-company-organization-staff/34564?u=jsick This should alleviate concerns that DM staff are being prevented from fully using the forum. This ticket implements a small notebook to exercise the Discourse API to make this trust level migration possible.",1 DM-4099,"Provide upstream improvements to sphinx-prompt","Provide PRs to sphinx-prompt (or decide to own a fork of sphinx prompt in documenteer) that includes - an actual package you can import - better error reporting when you forget to include a class with the prompt directive",1 DM-4100,"Replace use of image <<= with [:] in python code","Replace all use of the afw image pixel copy operator {{<<=}} with {{\[:]}} in Python code. See DM-4102 for the C++ version. These can be done independently.",2 DM-4102,"Remove use of <<= from C++ code in our stack","Replace usage of deprecated Image operator {{<<=}} in C++ code with {{assign(rhs, bbox=Box2I(), origin=PARENT)}} as per RFC-102 Switch from [:] to assign pixels in Python code where an image view is created for the sole purpose of assigning pixels (thus turning 2-4 lines of code to one and eliminating the need to make a view).",3 DM-4105,"Update user documentation","{{ORDER BY}}, {{objectId IN}} and {{objectId BETWEEN}} predicates support have been improved, this should be documented. ",1 DM-4112,"doc & demostrate ceph setup",NULL,3 DM-4113,"add lfs remote support to lsstsw/lsst_build","Support for cloning from lfs backed repos, when indicated via repos.yaml, is needed. ",4 DM-4115,"Update cfitsio to 3.37 (adding bz2 support)","Per RFC-105, we should upgrade to cfitsio 3.37.",2 DM-4116,"evaluate git lfs prototype and provide feedback",NULL,3 DM-4117,"Clean up lsst_stack_docs for preview","Improve the presentation of the New docs overall: # Add a Creative Commons license # Remove stub documents from the presentation # Put READMEs in all doc directories to explain what content will go in them # Clean up and update the source installation guide to reflect 11_0",1 DM-4118,"Local LSST IAM meeting","October 14, 2015 (local) - Drawing identity access management architecture on whiteboard. - DB access via kerberos - mariaDB does pam but does it do kerberos tickets? - could then simply access with a ticket - if users are exporting VMs/containers, do they need keytabs? how do we support this? - what does the ID linking? - do we need replication to base site so that base site can operate independent - Can we get 2 LSST VMs? - NCSA cyber-infrastructure standards? MIT Kerberos? OpenLDAP? - Meeting with Iain Goodenow ",2 DM-4119,"Security Meeting with LSST PO","October 8, 2015 - Refresh of security plan, sub-plan - Talk about camera subsystem refusal to buy-in to security plan - IaM work moving along - Continue bi-weekly meetings with developers - Most likely going with kerberos - Iain meeting with Jim and Co on authn/z work - Joint technical meeting in February - when and where? Santa Cruz, Feb 22-24th 2016 - security plan refresh: - cover email with old document and instructions - DM: Don P. and Jeff Kantor - EPO - PO: Iain - Camera - Incidents: all reports are collected and acknowledged",1 DM-4120,"Build instance and snapshot on Nebula for HTCondor worker with v11_0 LSST stack","Build instance and snapshot on Nebula for HTCondor worker with v11_0 LSST stack.",12 DM-4121,"Draft IaM Recommendations","LSST IaM draft recommendations from NCSA. Group includes: Jim Basney, Terry Fleury, and Dan Thayer.",23 DM-4122,"ctrl_events test failures on CentOS7 VM on Nebula","I am seeing failure to build due to test failures for ctrl_events on a CentOS7 instance on the Nebula Openstack. Details to follow.",1 DM-4123,"Bootcamp meeting","Slides and attendance at DM Bootcamp",8 DM-4124,"Bootcamp meeting","Travel to Princeton and attend DM Boot Camp ",8 DM-4125,"pipe_tasks/examples/calibrateTask.py fails","The self contained example calibrateTask.py in pipe_tasks/examples/ fails when attempting to set field ""coord"" in refCat. Exact error message - {code} 11:04:19-vish~/lsst/pipe_tasks (u/lauren/DM-3693)$ examples/calibrateTask.py --ds9 calibrate: installInitialPsf fwhm=5.40540540548 pixels; size=15 pixels calibrate.repair: Identified 7 cosmic rays. calibrate.detection: Detected 4 positive sources to 5 sigma. calibrate.detection: Resubtracting the background after object detection calibrate.initialMeasurement: Measuring 4 sources (4 parents, 0 children) Traceback (most recent call last): File ""examples/calibrateTask.py"", line 150, in run(display=args.ds9) File ""examples/calibrateTask.py"", line 119, in run result = calibrateTask.run(exposure) File ""/home/vish/lsst/lsstsw/stack/Linux64/pipe_base/11.0-2-g8218aaa+5/python/lsst/pipe/base/timer.py"", line 118, in wrapper res = func(self, *args, **keyArgs) File ""/home/vish/lsst/pipe_tasks/python/lsst/pipe/tasks/calibrate.py"", line 478, in run astromRet = self.astrometry.run(exposure, sources1) File ""examples/calibrateTask.py"", line 90, in run m.set(""coord"", wcs.pixelToSky(s.getCentroid())) File ""/home/vish/lsst/lsstsw/stack/Linux64/afw/11.0-5-g97168e0+1/python/lsst/afw/table/tableLib.py"", line 2372, in set self.set(self.schema.find(key).key, value) File ""/home/vish/lsst/lsstsw/stack/Linux64/afw/11.0-5-g97168e0+1/python/lsst/afw/table/tableLib.py"", line 1064, in find raise KeyError(""Field '%s' not found in Schema."" % k) KeyError: ""Field 'coord' not found in Schema."" {code} Note that {{wcs.pixelToSky(s.getCentroid())}} is set to {{Fk5Coord(15.007663073114244 * afwGeom.degrees, 1.0030133772819259 * afwGeom.degrees, 2000.0)}}",1 DM-4127,"Mask overlay does not work when the image is flipped",NULL,2 DM-4129,"Database connection problems in daf_ingest","The DbAuth connection fallback in ingestCatalogTask passes the ""password"" keyword argument to {{MySQLdb.connect}} instead of ""passwd"", which fails. Also, the ""port"" command line argument isn't marked as an integer, causing port strings to be passed down to MySQLdb. This results in a type error. ",1 DM-4131,"Have a ""mark as current"" option in lsstsw","Russell explained to me the advantage of having a specific eups tag associated with a given lsstsw installation. However, it would be very handy to have a way to get all installed packages automatically tagged as current as part of the installation process. I suggest a ""-c"" option to lsstsw which will tag everything as current after the full installation is complete. This way, partially installed packages won't get marked current, and people who do full installations can not have to deal with having to say ""-t bBLAH"" every time they setup things.",4 DM-4133,"Change type of LTV1/2 from int to float when writing afw images to FITS","The LTV1/2 problem is originally my bug. I used integer LTV1/2 in {code} afw/src/image/ExposureInfo.cc: data.imageMetadata->set(""LTV1"", -xy0.getX()); afw/src/image/ExposureInfo.cc: data.imageMetadata->set(""LTV2"", -xy0.getY()); {code} whereas a more careful reading of the NOAO page [http://iraf.noao.edu/projects/ccdmosaic/imagedef/imagedef.html] introducing them includes floating point examples. The fix is to cast the XY0 values to float. I'm not sure if there'll be any side effects of fixing this, but if so they'll be obvious and trivial. ",1 DM-4135,"DHT prototype: HTTP server library refactor/cleanup","This experimental library turned out to be quite useful. Next stage of prototyping will be making greater use of this, and I anticipate this library will also be used in production code. Spend some time cleaning up and organizing this lib so it doesn't get off to a bad start, and prepare for general review/feedback from the team.",6 DM-4136,"DHT prototype: test fixture rework","The DHT test fixture, developed during the preliminary work with kademlia, needs some updates for the next stage of the prototype: * adapt to reworked http library * adapt DHT interface to be more generic ",6 DM-4137,"Update DECam CCDs gain, read noise, and saturation values","The values of DECam gain, read noise, and saturation value need to be updated. This ticket is to update them in the Detector amplifier information, which is used in IsrTask. Talked to Robert Gruendl. These values should take precedence over the values in the fits header. They seem stable and do not seem to vary with time. ",1 DM-4138,"Re-implement packed keys in CSS","Current implementation of JSON-packed keys in CSS has one complication - the name of the container for packed keys is the same as the key itself (plus .json suffix). This complicates handling of the subkeys because these keys need to be filtered out and these names are all different. It would be better to have easily identifiable packed key names.",4 DM-4142,"Discover how to create Doxygen XML in build system","We need XML output from Doxygen in order to import existing API documentation into Sphinx. In this story I find out how to achieve this within the stack build system.",2 DM-4143,"Demonstrate using Breathe for Python & C++ API reference in New Docs","Demonstrate use of breathe for utilizing the existing Doxygen API documentation in the new Sphinx-based doc platform.",3 DM-4145,"Reduce scons output in qserv","Yesterday AndyH expressed a valid concern that qserv prints too much info which makes it hard to find errors. By default scons prints whole command line for C++ compilation and linking which are quite long (~half screen depending on your screen size). Most of the time we don't need to see that, so it would be better to replace that with shorter messages like ""Compiling Something.cxx"" and have an option to print full command with --verbose option.",1 DM-4146,"Networking requirements for design","Meeting with Paul to discuss remaining network design.",1 DM-4147,"Finance Contract Discussions","Discussing updated contract hoops and game plan.",1 DM-4148,"ICI agenda and mtg",NULL,1 DM-4150,"Investigating procurement of individual components","Component breakdown and explanation of components as part of FY16 purchases in order to plan optimal purchasing through vehicles available to NCSA.",1 DM-4151,"Search for uses of current afw.wcs in the stack","Search through the stack for all the uses of our Wcs implementation (Wcs, TanWcs, makeWcs, and any other hidden objects) and make a list of all of those uses (on Community for example). This list should note whether the usage is in C++ or python.",2 DM-4152,"Document detailing usage of Wcs in the stack","The information from DM-4151 should become a brief report on the kinds of usage of Wcs in the stack. This could be posted to Community or Confluence, or it could be a brief LaTeX document attached to the afw repo. Included in this report should be whether each current usage requires C++, or whether it could be done with e.g. vectors returned from python.",4 DM-4153,"RFD to collect current and future use cases of Wcs","File an RFD requesting information about current and possible future use cases for a Wcs system in the stack. This should get feedback from, at minimum, the alerts pipeline, DRP, and Level III data producers. It should also get feedback from our resident wcs experts. Whether those use cases are currently implemented in our code, or could be generalized from it, isn't important to this RFD as this information will feed into a subsequent requirements document and RFC about what needs to be written/changed. Some use cases/buzzwords that will likely be included: * Efficient x,y -> x',y' * Stacking a sequence of transformations efficiently * easy extensibility * should color terms be included in the Wcs or dealt with elsewhere? * pixel distortion effects (e.g. tree rings, edge roll off) * Simultaneous Astrometry (i.e. from image stacks)",4 DM-4154,"Sever side Histogram for variable bin size","For any given column or columns (expression of columns such as col1+log(col2) ) of a IpacTable data, the variable bin histogram is needed. The variable bin is based on ""Bayesian Blocks"" algorithm (http://jakevdp.github.io/blog/2012/09/12/dynamic-programming-in-python/). The output is a new IpacTable (DataGroup) with three columns, numPoints, binMin and binMax.",6 DM-4155,"LSST Wcs requirements document","Based on the information compiled from DM-4153 and DM-4152, prepare a requirements document (LaTeX, with references) describing all of the known Wcs requirements for the various portions of the LSST stack. This document could live in afw or in its own repository, and could potentially become a published technical report/conference proceeding.",10 DM-4156,"Evaluate existing Wcs libraries and report on our options","As part of the requirements document in DM-4155, we need a report-likely section(s) of that same document-on the currently available Wcs libraries (including, but not limited to AST Starlink and astropy.Coordinates) and summaries and references from the current literature about how other surveys and projects have managed their Wcs. For each of the currently existing options, the report should include at a minimum: * implementation language. * API and languages that can interface to that API. * how well supported (including number of contributors) is the project and how active is ongoing development. * its performance and optimization for both scalar and vector calculations, and its scalability to large data sets. We also need to look for and/or request technical reports and other literature from existing and completed surveys, including DES (Robert Gruendl), Pan-STARRS (Paul Price), STScI (Erik Tollerud), and SDSS (Lupton and/or Blanton?).",30 DM-4157,"Provide a recommendation for how to manage Wcs in LSST","The technical report from DM-4155 and DM-4156 should have as its conclusion a recommendation for what Wcs system the LSST stack should adopt (either our own implementation, a third-party library, or some combination thereof). Beyond the conclusion section of that report, this will be provided as an RFC that includes: * executive summary * mock API * links to the relevant documentation (beyond just the above technical report) * a bullet list of the rationale for this decision The conclusion of this RFC represents the end of this Epic.",4 DM-4160,"Unused variables in meas.algorithms.utils","Pyflakes 1.0.0 reports: {code} $ pyflakes-2.7 utils.py utils.py:232: local variable 'chi2' is assigned to but never used utils.py:481: local variable 'numCandidates' is assigned to but never used utils.py:482: local variable 'numBasisFuncs' is assigned to but never used utils.py:487: local variable 'ampGood' is assigned to but never used utils.py:492: local variable 'ampBad' is assigned to but never used {code} In the best case, those variables are simply unnecessary, and they should be removed to simplify the code and avoid wasting time. Alternatively, it's possible that they ought to be used elsewhere in the calculation but have been omitted accidentally. Please establish this for each one, then either remove them or fix the rest of the code.",1 DM-4162,"Please implement a warper that works with a single XYTransform","At present we only warp images based on a pair of {{Wcs}}. This is needlessly restrictive. We should be able to define the transformation by function that computes {{f(x,y) -> x',y'}}, e.g. an {{XYTransform}}. Note that reversibility, while not strictly necessary, is very desirable. Hence we might as well use {{XYTransform}}. I suggest we have only one underlying implementation in order to avoid code duplication. This could easily be done by implementing an {{XYTransform}} that combines a pair of WCS.",6 DM-4164,"Create a skeleton framework for Firefly using redux and react.","As part of the GWT conversion to pure JavaScript, we've came up with a new design based on redux and react. This task is to create all of the major components of the framework with minimal functionalities. This will allow other developers to build upon this foundation core.",10 DM-4165,"Take upstream boost 1.59 patch to squelch warnings for gcc 5.2.1","Under gcc 5.2.1, use of boost 1.59.0 produces a torrent of compiler warns from within boost headers about use of deprecated std::auto_ptr (see https://svn.boost.org/trac/boost/ticket/11622). A patch for this is already committed upstream in boost. It is proposed that we take this patch into the lsst t&p in interim until the next official boost release.",1 DM-4166,"Internal documentation of procurement process",NULL,1 DM-4168,"productize ""Data repository selection based on version""","finish & productize work from DM-5608",6 DM-4170,"Butler: move configuration (.paf) file into repository","We want to be able to keep the .paf file in the repository, with the data for which it provides configuration information. At least for the time being it needs to be optional; it looks first in the repository and second in the ""old"" location for the paf. In the case of a repository chain, the child should 'win'.",12 DM-4171,"Butler: change configuration from .paf to something else","for ""something else"" there are 2 major options: pex_config (dislike because the 'data' gets executed), and yaml (very well supported, is just data). 3rd option: we could use the sqlite registry, and define a schema for that. (sqlite benefits: has support for write locks that will be necessary very soon. Except, need write-once-compare-same mechanism; so we'll need that 1. for normal files and 2. across n nodes. so maybe there's no benefit). also, while we're there: as noted in DM-4170, registries.py needs to not use astropy and instead should use pyfits. If it's quick, that change should be made here (or turned into a separate ticket)",12 DM-4172,"Build AL2S vlans from Miami to NCSA","To prototype the layer 2 circuit LSST will eventually have.",2 DM-4174,"document proposal for Base site to NCSA data transfer","With Steve and James distill the actual data movement requirements, the mechanism to broker those transfers and the network technology to ensure the real-time transfer of images",2 DM-4175,"Provide network infrastructure support to deployment of new nodes",NULL,3 DM-4176,"Create baseline requirements for evaluating server hardware from different vendors",NULL,1 DM-4177,"Refine server evaluation specifications into a usable quantifiable form",NULL,1 DM-4178,"Discussions with Ron on Base site architecture",NULL,1 DM-4179,"update LSE-78 once current updates are applied","There are several sections with inaccurate information about the North American portion of the LHN and the implementation of the networking into NCSA as well as the base site commissioning cluster architecture.",2 DM-4180,"Butler: provide API so that a task can define the output dataset type","The task needs to be able to specify everything that needs to be in the policy file so that the butler can put and get data for a new dataset type. Consider that the policy data can be split between the camera-specific and the task-specific parts. (KT was thinking of calling the camera part the ""genre""), this potentially reduces the amount the task has to specify. Another option is to hard-code some some of the policy in the butler itself: * the path template (it could be assembled out of the data id components) ** if it's hard coded the task must pass the component dataId keys ** if it's not hard coded the task must provide a template * python type * storage type there could be user overrides too.",20 DM-4181,"Butler: add support for skymap based dataIds","load the skymap from the repository and use it a la metadata lookup to complete lookups. for example if the script wants all the patches that are in tract x, look in the skymap to get that information. will be something like: create a skymap object given a configuration (in the repository) (it has its own dataset type) create a registry with that skymap object use that registry to lookup skymap related parameters. Might need to add to the policy file that a given configuration uses the skymap. requires that data is already indexable by tract & patch, the work is adds iteration over tract/patch specified by the skymap. example use case: {code} { datasetType = 'coaddTempExp' key = (?) format = ['patch', 'visit'] dataId = {'tract':} myButler.queryMetadata(datasetType, key, format, dataId) } {code}",15 DM-4183,"Experiment with memcached for secondary index","Test whether memcached could be used to serve objectId --> chunkId mapping, in particular from the performance perspective.",6 DM-4184,"Experiment with xrootd for secondary index","Test whether xrootd could be used to serve objectId --> chunkId mapping, in particular from the performance perspective.",6 DM-4185,"Experiment with c-style arrays for secondary index","Use simple C-style array (chunk[objID], allocated up to maximum number of objectIDs), or a single-layer set of arrays (chunk[block][objID%blksize], where second index runs from 0 to size of each block) to store index compactly and provide minimum overhead for lookups.",5 DM-4188,"Infrastructure Security at Site",NULL,1 DM-4189,"Outline of data flow","Outline of data flow between Chile->NCSA and NCSA->Chile.",3 DM-4190,"L1 system functions and responsibilities ","Outline functions and responsibilities for all parts of the L1 system",35 DM-4191,"Initial code change to run ISR with DECam raw data","This ticket is for implementing changes in {{obs_decam}} in order to run {{processCcd}} with raw DECam data. Changes are mostly in {{DecamMapper}} and a new class {{DecamIsrTask}} is added. A test to retrieve defects with Butler is also added. ({{testdata_decam}} is at lsst-dev /lsst8/testdata_decam/ ) The pixels with bit 1 (bad, hot/dead pixel/column) from the Community Pipeline Bad Pixel Masks are used as bad pixels. The CP BPM fits files are directly used as defect files. Due to their large size, they probably should not go into {{obs_decam}} repository so they are treated similarly as other calibration products. With the changes of this ticket, the following ingest defects files into {{calibRegistry}}: {code:java} ingestCalibs.py . --calibType defect path-to-bpm/*fits {code} and the following should run past ISR: {code:java} processCcd.py . --config isr.doFringe=False isr.assembleCcd.setGain=False calibrate.doPhotoCal=False calibrate.doAstrometry=False calibrate.measurePsf.starSelector.name=""secondMoment"" {code} Running fringe correction with DECam raw data will be in future tickets (DM-4223 and possibly more). Also this ticket does not cover implementing or porting new ISR functionalities that haven't yet been included in ip_isr (such as crosstalk). ",14 DM-4192,"Other LOE -- Oct 2015","Local LSST group meetings, Ethics training, or other local meetings or tasks to comply with NCSA policies",2 DM-4193,"Automate LSST Firefly standalone releases using Jenkins","This task involves merging in feature branches, building Firefly standalone, tagging, push changes to github, generating changlog, and using github API to publish the release. Release should be attached to the latest tag with downloading artifacts and changelog.",7 DM-4194,"Python LogHandler does not pass logger name to log4cxx","Not sure how or why it happened, but presently Python LogHandler for lsst.log does not pass logger name to log4cxx layer and all messages from Python logging end in root logger. ",1 DM-4195,"Build proof-of-concept package documentation for lsst.afw","Build package documentation for {{lsst.afw}} under the new doc platform as a demonstration. - Install a Sphinx site in lsst.afw/doc - Implement MVP documentation pages for lsst.afw packages (table, image, etc.) - C++ API reference from doxygen+breathe - Python API reference from numpydoc This ticket *will not* attempt to add new documentation content; only to show how existing content can be re-organized.",2 DM-4196,"Build ltd-mason for running a multi-package software documentation build","LSST the Docs ([SQR-006|http://sqr-006.lsst.io]) is a system for extending our existing Jenkins build infrastructure to build Sphinx-based software documentation for our Eups-based packages. This is a ticket to implement the {{ltd-mason}} service, which runs on Jenkins after the {{buildlsstsw.sh}} step and compiles software documentation. Specific outcomes of this ticket are - Full specification of the YAML interface between {{ltd-mason}} and {{buildlsstsw.sh}} (including creating a mock YAML file for local testing) - Demonstration of a Science Pipelines documentation build on a local lsstsw environment (in conjunction with content from DM-4195) - Accommodations to the science pipelines documentation repo and documenteer for building sphinx packages are included in this ticket’s scope. Next steps are - Standing up the service on Jenkins and testing integration with {{buildlsstsw.sh}} - Uploading to S3 (which involves building integration with {{ltd-keeper}}.",17 DM-4197,"Improve Qserv master robustness for queries like ""select @@max_allowed_packet""","This king of query crashes Qserv: {code:bash} mysql --host 127.0.0.1 --port 4040 --user qsmaster -e ""select @@max_allowed_packet"" {code}",1 DM-4201,"Documentation and technical debt in meas_base/PixelFlags.cc","The port from HSC of SafeClipAssembleCoaddTask has left some documentation and code quality changes to be made. Specifics include: * In the process of porting, functionality was added which allowed users to specify additional mask planes to be converted to pixel flags. However this was fundamentally incompatible with the flag handler functionality that LSST was currently using. PixelFlags was thus modified to allow SafeClipAssemble coadd to work with the user defined masks, but that made it fundamentally different than the other plugins. In the future this plugin should be brought more in line with all the other measurement framework. This will most likely involve rewriting sections of the measurement framework, to add the ability for users to more directly set runtime behaviors of the measurement plugins (such as specifying non default mask planes to work with). * Because PixelFlags could no longer use the the flag handler framework, sections of the SafeCentroidExtractor had to be duplicated within PixelFlags. This duplicated code is non-ideal and should be rectified. Possibly in the process of rewriting the measurement framework, the utils could be expanded to have convenience methods to access functionality when not using flag handlers * As with all of the measurement framework, PixelFlags needs better documentation. This includes some line to line comments, but more importantly the over all functionality of the routine needs documentation. This includes: what the routine does; how it works; and why various design decisions were made",3 DM-4202,"Revert temporary disabling of CModel in config override files","Revert the temporary disabling of CModel that relates to a bug noted in DM-4033 that was causing too many failures to test that processCcd (etc.) would run all the way to completion (most of the other fixes/updates related to the initial disabling in the multiband tasks have now been completed in DM-2977 & DM-3821). Relevant files: {code} config/processCcd.py config/forcedPhotCcd.py config/forcedPhotCoadd.py config/measureCoaddSources.py {code} ",1 DM-4206,"wmgr should delete database from inventory when dropping it","When wmgr drops database it should also cleanup chunk inventory for that database. ",2 DM-4208,"Research web authentication and authorization and gather usage stories","Research SUI, DAX, Butler, and Qserv authentication and authorization requirements and schemes. Document usage stories for all layers",6 DM-4209,"Create unit tests for SafeClipAssembleCoadd","Porting SafeClipAssembeCoadd from HSC to LSST left that functionality without a unit test. Currently AssembleCoadd is tested from within tests/testCoadds.py. This test does not call AssembleCoadd directly however. The actual code pertaining to assembling a coadd exists within python/lsst/pipe/tasks/mocks/mockCoadd.py. This called from testCoadds, and is used to build and coadd synthetic images from known psfs amongst other things. This should be expanded to test both AssembleCoadd and SafeClipAssembleCoadd, possibly with some sort of argument to the function call. It is important to keep testing both methods of generating a coadd.",3 DM-4210,"Create documentation and examples for SafeClipAssembleCoadd","SafeClipAssembleCoadd in HSC did not have adequate documentation, and thus neither does LSST post port. Documentation which details the functionality and usage of this function should be created, and should be available either through the [Doxygen task documentation|https://lsst-web.ncsa.illinois.edu/doxygen/x_masterDoxyDoc/group___l_s_s_t__task__documentation.html] or its successor . Examples of the usage and various options should also be included with the documentation.",4 DM-4212,"Investigate shear bias errors from DM-1135","DM-1135 mostly was inconclusive, due to the fact that the error bars were around the same size as the differences in many of the tests. This in spite of the fact that we ran 6x as many sample galaxies as great3sims. Investigate the reason for this, and see if we can estimate the errors more accurately. ",6 DM-4213,"Rerun tests of DM-1135 with a larger number of galaxies","As a result of the new error estimation, it became apparent that a larger number of sample galaxies were required in DM-1135. This is an expansion of that test to a larger number of galaxies. Since the original great3 tests had about 2 million galaxies, we should be able to do these test reasonably well with the 6 million galaxy pool created in DM-1135. However, the measurements need to be rerun in some cases, and the error analysis done again.",6 DM-4214,"Initial tests of ShapeletApprox","This will be part of a wider set of test which I am hoping that Jim will fill in as additional stories under DM-1136 Basically, we hope to see how extensive a Shapelet Approximation must be done for a Psf in order to reach a stable result (stable meaning it would not markedly improve with increase in Shapelet order). For this intial test, I propose to compare SingleGaussian, DoubleGaussian, Full, and 2 higher order models. I will also add a model with inner and outer defined to see if those have a significant effect. It will tell use how the existing models line up and which parameters matter. This test will be done using the great3sims subfield organization, and with a single, randomly chosen Psf image from the 0.7 arcsec FWHM library (raw fwhm from PhoSim, the actual fwhm is closer to 0.8 arcsec) This will probably be just the beginning of more extensive ShapeletApprox tests, so this story should be a sub-story of a larger test project which I am hoping Jim will define.",6 DM-4215,"Butler: SafeFile and SafeFileName can overwrite good with bad in some cases","A bad file can be written in Butler, in the case where 2 temp files that use SafeFile or SafeFileName to the same location are started, one closes, and then the other fails - and then closes and writes the bad file. Need to handle the exception in a way that does not write. Also, in the case of no exception (failure), when closing B, need to compare to A and throw if different.",4 DM-4219,"Package capnproto for eups","Prototype is now getting to the point where a wire-protocol package like capnproto or protobuf is needed. capnproto is the new hotness, and we're probably going to want to migrate qserv from protobuf->capnproto at some point. This task is to go ahead and get capnproto packaged and published for use in the replication prototype.",2 DM-4220,"Convert copyright/license statements to one-liners for RFC-45","Refactor how we manage copyright and license information in stack repositories # Identify a list of packages to process # build and test an automated system of - adding a global COPYRIGHT to each repo. Content will be “Copyright YYYY-YYYY The LSST DM Developers”. Years will be determined by git history. - adding a GPLv3 license file to each repo. - changing the boilerplate in all files to say ""See the COPYRIGHT and LICENSE files in the top-level directory of this package for notices and licensing terms.” Use https://gist.github.com/ktlim/fdaea18ab3d39afdfa8e - automatically branch, commit, merge and push And deploy this automated system.",8 DM-4222,"X16 Secondary Index - Implementation","Implement / productize optimizations to secondary index proposed through DM-2119",27 DM-4223,"IsrTask calls removeFringe in FringeTask but the method does not exist","The method {{removeFringe}} of {{FringeTask}} is called in {{IsrTask}} but there is no {{removeFringe}}. Not sure if {{removeFringe}} was meant to be a place holder",1 DM-4224,"getExposureId not implemented in obs_lsstSim","There is no implementation of the getExposureId method in processEimage.py. This causes it to fail using a modern stack.",1 DM-4225,"Collect single-host performance data for secondary index","Run production-scale (billions of entries) tests on different index options, collect performance statistics for allocation (CPU, memory) and for queries.",3 DM-4226,"Set up multi-host tests for secondary index technologies","For client-server technologies (memcached, xrootd, etc.), develop multihost test jobs to exercise production-scale indices (billions of entries, millions of queries). Index should be allocated on single ""master"" machine, with queries generated from one or more separate machines, possibly with multiple threads/jobs per machine.",5 DM-4227,"Experiment with bulk updates to secondary index","The nightly data loader job may need to add new objectIDs, or change the chunks of existing objectIDs, _en masse_. Develop test code (or add features to existing demonstrators) to handle bulk loading of new data to the index, or to overwrite collections of existing data. This is significant for client-server technologies, where the bulk data may be transferred in a single transaction, with the server (possibly) handling the loop over individual elements.",7 DM-4228,"Collect multi-host and bulk-update performance data for secondary index","Run secondary index tests across multiple hosts (server and clients), collecting performance data for production-scale indices (billions of entries, millions of queries) with many parallel queries and bulk/block updates.",5 DM-4229,"Identify candidate technology for secondary index","Evaluate results of production-scale performance tests, both single and multiple host. Identify the technology most likely to meet requirements, and estimate performance capability with respect to those requirements",3 DM-4230,"Port HSC-1355: Improved fringe subtraction","[HSC-1355|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1355]: ""with this fix, we get much better fringe subtraction"".",2 DM-4231,"Data Distrib proto (nov)",NULL,20 DM-4232,"Variance is set after dark subtraction","In the default {{IsrTask}}, the variance is currently set after dark subtraction. This means that photon noise from the dark is not included in the variance plane, which is incorrect. The variance should be set after bias subtraction and before dark subtraction. [~hchiang2] also points out (DM-4191) that the {{AssembleCcdTask}} with default parameters requires amplifier images with variance planes, even though the variance cannot be set properly until after full-frame bias subtraction. I believe that {{AssembleCcdTask}} only requires a variance plane in the amp images because it does an ""effective gain"" calculation, but I suggest that this isn't very useful (an approximation of an approximation, and you're never going to use that information anyway because it's embedded in the variance plane with better fidelity). I therefore suggest that this effective gain calculation be stripped out and that {{AssembleCcdTask}} not require variance planes.",5 DM-4234,"HSC backport: Jacobian and focalplane algorithms","Add algorithms to compute the *Jacobian* correction for each object (calculable from the Wcs, but sometimes convenient) and record the *focal plane* coordinates (instead of CCD coordinates) for sources (useful for plotting). The standalone HSC commits to be cherry-picked are: *Jacobian* {{meas_algorithms}} May 3, 2013 [Jacobian: add Algorithm to compute the Jacobian.|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/88d3bd3f32cf4d0138b80148e57bc275fc8c3454] May 24, 2013 [Jacobian: fix up some cut/paste oversights.|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/ecad0d2559bb9815fc5560234f4502f35f50db73] May 28, 2013 [Jacobian: fix calculation|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/7f3db53b56279929b9e416173ed09cf00dc81406] {{obs_subaru}} May 3, 2013 [config: enable jacobian calculation in processCcd|https://github.com/HyperSuprime-Cam/obs_subaru/commit/d0969911ee1a655fd82998f0b936fa90f443d2fd] May 6, 2013 [config: set pixelScale for jacobian correction|https://github.com/HyperSuprime-Cam/obs_subaru/commit/e36bd1b4410812ca314f50c01f899d92acc0e7a5] *focalplane* {{meas_algorithms}} May 24, 2013 [add algorithm to calculate position on the focal plane|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/dda3086f411d647e1a3e15451d7f093cd461873a] May 25, 2013 [fix up building of focalplane algorithm|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/57d718bf51b255adf5789e389dfb776ecaa062d1] Nov 21, 2014 [Adapt to removal of Point from afw::table.|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/95627d55cb7d64718a42027954474df5c3661a65] {{obs_subaru}} May 24, 2013 [config: activate focalplane algorithm|https://github.com/HyperSuprime-Cam/obs_subaru/commit/d999a32e7e10b25cceccc94b61890486f96c0bfd] ",4 DM-4235,"HSC backport: countInputs and per object variance functions","Back port of the following two HSC tickets: *countInputs* [HSC-1276|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1276] {{meas_algorithms}} Jul 1, 2015 [add measurement algorithm to count input images in coadd|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/51db0fd2624c7f9b641c93aa3cf6366539995d50] {{obs_subaru}} Sep 24, 2015 [config: activate countInputs for measureCoaddSources.|https://github.com/HyperSuprime-Cam/obs_subaru/commit/13ecd1317b05b5ff9e65fba41fe27a5cffcc2fda] *variance* [HSC-1259|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1259] {{meas_algorithms}} Jul 2, 2015 [add measurement algorithm to report background variance|https://github.com/HyperSuprime-Cam/meas_algorithms/commit/86022f4381c3cec3f7f203b831b6a306596cfa3f#diff-7ae7aea69b58dbf075350ccfd3802cfb] {{obs_subaru}} Oct 19, 2015 [config: activate measurement of variance for coadds|https://github.com/HyperSuprime-Cam/obs_subaru/commit/cf1e80958bb9164dacf42d2d35a94dd366c78892] ",6 DM-4236,"Specify default output location for CmdLineTasks","When neither {{\--output}} or {{\--rerun}} is specified as an argument to a {{CmdLineTask}}, any output from that task appears to be written back to the input repository. Note the use of the term ""appears"": from a preliminary inspection of the code and documentation, it's not clear if this behaviour can be overridden e.g. by environment variables. The HSC stack behaves differently, using {{$INPUT/rerun/$USER}} as a default output location. A [brief discussion|https://community.lsst.org/t/new-argument-parser-behavior-rerun-flag-introduction-discussion/345] suggests that this is the preferred behaviour. Please update the LSST stack to match the HSC behaviour.",2 DM-4237,"unable to upload images to nebula","I seem to be unable to upload an image to neblua from a URL via either horizon or the nova cli client. The request seems to queue briefly and then reports a status of {{killed}}. Eg {code} glance image-create --name ""centos-7.1-vagrant"" --disk-format qcow2 --container-format bare --progress --copy-from http://sqre-kvm-images.s3.amazonaws.com/centos-7.1-x86_64 --is-public False --min-disk 8 --min-ram 1024 {code}",2 DM-4238,"Fix integer casting error in numpy version 1.10 in obs subaru","Fix type casting in obs_subaru in lates numpy in obs_subaru",1 DM-4239,"Identify Qserv areas affected by secondary index","Evaluate Qserv software for the Czars and workers to identify where an interface to the secondary index will be required for efficient operation.",5 DM-4240,"Implement secondary index service in Qserv","Implement the selected seconary index technology (see DM-2119) as a Qserv service, providing a client API to be used elsewhere in the Qserv system. Depending on the chosen technology, this may including configuration of a separate lightweight server for the secondary index, creation of database tables, etc. The client-side API should be technology independent.",10 DM-4241,"Implement bulk updating of secondary index in Qserv","Provide a service in Qserv to support creation or modification of secondary index objectID-chunk pairs in bulk, to support data loading.",4 DM-4242,"Implement secondary index query in Qserv","Implement query of secondary index in Qserv, using the technology selected in DM-2119.",4 DM-4243,"Draft of Configuration Solicitation","Draft of solicitation to be used for quote and configuration from multiple vendors for FY16 purchase. ",1 DM-4245,"Image Viewer memory leak","When reloading the same 500MB RAFT image into an image viewer (see the script below), it was discovered that single node Firefy server with 3G memory runs out of memory after ~15 reloads Test case: keep reloading the html file with the following Javascript, creating an image viewer with 500MB image: function onFireflyLoaded() { var iv2= firefly.makeImageViewer(""plot""); iv2.plot({ ""Title"" :""Example FITS Image'"", ""ColorTable"" :""16"", ""RangeValues"":firefly.serializeRangeValues(""Sigma"",-2,8,""Linear""), ""URL"" :""http://localhost/demo/E000_RAFT_R01.fits""}); } Follow up: The bug was traced to java.awt.image.BufferedImage objects not being evicted from VIS_SHARED_MEM cache. Further search showed that java.awt.image.BufferedImage (along with java.io.BufferedInputStream) is in src/firefly/java/edu/caltech/ipac/firefly/server/cache/resources/ignore_sizeof.txt, which lists the classes that have to be ignored when calculating the size of cache. Testing on single node server (VIS_SHARED_MEM cache is not replicated), using [host:port]/fftools/admin/status page: BEFORE (java.awt.image.BufferedImage was commented out in ignore_sizeof.txt) After 14 reloads: Memory - Used : 3.7G - Max : 3.55G - Max Free : 488.0M - Free Active : 488.0M - Total Active : 3.55G Caches: VIS_SHARED_MEM @327294449 Statistics : [ Size:15 Expired:0 Evicted:0 Hits:246 Hit-Ratio:NaN Heap-Size:1120MB ] OUT OF MEMORY on next reload AFTER THE CHANGE (Commented java.awt.image.BufferedImage in ignore_sizeof.txt) After 36 reloads: Memory - Used : 1672.9M - Max : 3.55G - Max Free : 1968.0M - Free Active : 1468.0M - Total Active : 3.6G Caches: VIS_SHARED_MEM @201164543 Statistics : [ Size:3 Expired:0 Evicted:34 Hits:659 Hit-Ratio:NaN Heap-Size:1398MB ] ",2 DM-4246,"Local LSST Sec Meeting","Local cyber security meeting at NCSA with DM group.",2 DM-4247,"Image viewer: choosing pixel interpolation algorithm for scaled images","Pixel values are defined at integer coordinate locations. This means that when an image is rendered in a scaled, rotates, or otherwise transformed coord. system, an interpolation algorithm should be used to provide a pixel value at any continuous coordinate. Currently, Firefly is using RenderingHints.KEY_INTERPOLATION = RenderingHints.VALUE_INTERPOLATION_NEAREST_NEIGHBOR, which means that when an image is rendered in a transformed coord. system, the pixel value of the nearest neighboring integer coordinate sample in the image is used. ""As the image is scaled up, it will look correspondingly blocky. As the image is scaled down, the colors for source pixels will be either used unmodified, or skipped entirely in the output representation."" Jon Thaler's team would like to be able to choose a different interpolation algorithm, depending on the situation. As an example see various resize algorithms in [thttp://stackoverflow.com/questions/4756268/how-to-resize-the-buffered-image-n-graphics-2d-in-java].",4 DM-4248,"LSST PO Security Meeting","Bi-weekly meetings with PO to discuss cyber security issues in LSST.",1 DM-4251,"Please include obs_subaru in CI","{{obs_subaru}} should be included in the CI system.",1 DM-4252,"Create GitLFS Technical Note","Create a SQuaRE Technical Note describing the architecture of the GitLFS service implementation.",2 DM-4253,"Literature search for DCR -- Reiss","Go through the literature to find relevant seminal papers on DCR. The outcome will be a bibliography and executive summary. This should be posted on Discourse.",15 DM-4254,"Implement simple reference index files","We would a very simple way to make small reference catalogs for astrometry and photometry. The use case is testing and small projects, where the overhead of making a full up a.net index file (or whatever we replace that with) is excessive.",5 DM-4255,"Determine scope and requirements for CalibrateTask redesign","There are two stories in DM-3579 that define two requirements for the redesigned CalibrateTask. Those along with the requirement that CalibrateTask be less brittle are a starting point for what the new task needs to do. All other requirements should be gathered using the RFC process. Using the input from the RFC and other requirements, the scope for this particular redesign will be defined and stated in a discourse thread.",10 DM-4256,"Sphinx support of sqr-001 technical note","Support the distribution of a technical note SQR-001 - Remove oxford comma in author list (documenteer) - Solve issue where title is repeated if the title is included in the restructured text document - Solve issue where name of the HTML document is README.html",1 DM-4258,"Investigate current dipole measurement examples and tests","There are tests in ip_diffim/tests/dipole.py for the dipole fitting. There is also an example in the examples directory of ip_diffim. This story will investigate these tests and examples to see to what precision the tests go as well as how complete the tests are. An outcome of this will be an understanding of how precise the tests need to be to show that dipole measurement is behaving as we expect.",4 DM-4259,"Create a set of tests (or update the current ones) to facilitate refactoring of dipole measurement","This will create a test (not necessarily a unit test) that will simulate dipoles and measure them so that the measurement can be compared to truth values. This may be simply refactoring the current tests. This task should also include generating more generalizable utilities needed to create the dipoles and incorporating these and other test data into the stack so that they can be used in other studies.",5 DM-4261,"F17 Integrate L1 Database with Alert Production pipeline","Integrate the prototype of the L1 database built through DM-2036 with the Alert Production pipelines. The design of the Alert Production L1 database is covered [here|http://ldm-135.readthedocs.org/en/master/#alert-production-and-up-to-date-catalog].",79 DM-4262,"Integrate qserv docs into the new doc system","See Frossie 11/04/2015 email to qserv-l list: {{5- Docs. So you guys have a sphinx site. Fab, that’ll make it super easy to drop it into the new doc system that hosts sphinx on readthedocs - you can see a small example here http://sqr-001.lsst.codes/en/master/ - the idea is to continuously deploy the docs so the release step should be very lightweight and doable via API calls. (My holy grail is to do a release with no local checkout involved). }}",2 DM-4263,"SQuaRE design meeting 1","Hold an in-person design discussion with members of the SQuaRE team. ",4 DM-4264,"SQuaRE supertask design meeting 2","Hold a teleconference design discussion with members of the SQuaRE team. ",1 DM-4265,"Create SuperTask design proposal","Create a design document covering the initial SuperTask concept and a scheme for how it would lead to solutions to longer-term pipeline construction problems, support provenance recording, etc.",16 DM-4266,"Should only read fringe data after checking the filter","The fringe subtraction is not necessarily performed if {{doFringe}} is True. It is only if the filter of the raw exposure is listed in config fringe.filters. Fringe data should not be read unless the filter is indicated. There are likely no such filter data and it would cause runtime errors. Seems related to changes from RFC-26 and DM-1299. ",8 DM-4267,"Adapt an existing task to be usable as a SuperTask","Either by wrapping, or by converting, make an existing task usable as a SuperTask subclass so that it can be run under an Activator.",8 DM-4268,"FY20 Define Procedures and Build Tools for Schema Evolution","As described in [LDM-135|http://ldm-135.readthedocs.org/en/master/#data-production-related-requirements], The database system must allow for occasional schema changes for the Level 1 data, and occasional changes that do not alter query results for the Level 2 data after the data has been released. This epic involves preparing for dealing with schema changes in production syste.",79 DM-4269,"Week end 10/03/15","Support for lsst-dev cluster, OpenStack, and accounts for week ending October 3, 2015.",1 DM-4270,"Week end 10/10/15","Support for lsst-dev cluster, OpenStack, and accounts for week ending October 10, 2015.",4 DM-4271,"Week end 10/17/15","Support for lsst-dev cluster, OpenStack, and accounts for week ending October 17, 2015.",3 DM-4272,"Week end 10/24/15","Support for lsst-dev cluster, OpenStack, and accounts for week ending October 24, 2015.",7 DM-4273,"Week end 10/30/15","Support for lsst-dev cluster, OpenStack, and accounts for week ending October 30, 2015.",3 DM-4274,"Planning for new equipment setup (week end 10/03/15)","* Planning for hardware upgrades in racks * Evaluate the setup of the new storage server installs lsst-store101, lsst-store141,lsst-store143,lsst-store144 * Set up and tested ESXi server on new Mac Pro. Worked on getting Mac OS X installed inside of ESXi on the new Mac Pro hardware. Found solution, need to find a better one if we are expecting to bring up and tear down instances on demand.",5 DM-4275,"New equipment setup (week end 10/10/15)","* Planning on how to start setting up for new equipment * Started to coordinate with Josh Hobblitt on what has been ordered and delivered * Received 15 Dell R730 servers (plan to set up in temporary rack before Thursday’s maintenance) * Installed spare rack at the east end of NCSA 3003-Row A. Added three 125V 30A drops to spare rack * Resolved problems with Puppet on lsst-stor101 * Updated OS on lsst-stor101, lsst-stor142-144 * Installed ZFS on lsst-stor101, lsst-stor142-144 * Checking config of stor142-144 and stor101 for Thursday outage * More ESXi testing with Mac Pro",6 DM-4276,"New equipment setup and regular maintenance (week end 10/17/15) ","* Unpack and mount 6 Dell R730’s * Update the bios and firmware on 6 Dell R730’s * Install ESXi 6.0 on 6 Dell R730’s * Rearranged hardware in NCSA 3003, move systems around in racks, reconnect power and networking, troubleshoot startup issues * Applied kernel and other software updates to infrastructure as it was moved * Setup vSphere vCenter server and compute nodes * Discovered need to order drive caddies for lsst10 and updated iDRAC Enterprise license for 15 new R730 servers",12 DM-4277,"New equipment setup and configuration (week end 10/24/15) ","* Unbox and mount six UPS in racks, mount two new power panel PDU's for 30 Amp service, connect power cabling to UPS and to each of the supported systems. TODO: Complete the setup on each of the servers. * Unbox and mount in the rack two R730 servers, connect network and power * Itemized list of new Dell servers received * Debugging networking issues on new lsst-esxi1 server - confirmed it's not an issue with switch ports or cables * Setup vSphere virtual networking and moved 4 test VMs to new setup * Setup vSphere Data Protection (Vmware backups via snapshots)",6 DM-4278,"New equipment setup and configuration (week end 10/30/15)","* Moved spare rack to Row C * Documented setup of new LSST vSphere setup (https://wiki.ncsa.illinois.edu/display/LSST/LSST+vSphere) * Debugging networking issues on new lsst-esxi1 server - testing with new/alternate hardware * Installed 6 new drives for historical log storage on lsst10 MySQL server",5 DM-4279,"bi-weekly IaM meeting","Meeting Oct 22nd Notes on LSST confluence",1 DM-4280,"Bootcamp meeting","IAM group attended DM bootcamp",4 DM-4281,"write meeting agenda, for Oct 19 meeting ",NULL,1 DM-4282,"Conduct, document, initial follow through on Oct JCC meeting ",NULL,1 DM-4283,"Visit to Argonne - facility coordination",NULL,2 DM-4284,"Specify L1 system - design and WBS plan for construction","Spent 3 days in focused planning to knock out version 1 of the L1 system plan. SPs cover me, Don.",12 DM-4286,"Processing of DECAM COSMOS field - Part I","This story covers work on the verification plan done in October. The DECAM Cosmos field was selected, however DECAM ISR is not available so the starting point for now is Community Pipeline reduced data. Have put the data through processCcdDecam and makeCoaddTempExp but had a number of failures that we have so far not been able to pin down. A list of user experience issues is being collated and [~frossie] will generate Summer 16 cycle stories to address those that fall within SQuaRE's defined scope of activities. Story closed to fit within month, but work is ongoing. ",20 DM-4287,"Software Stack Introduction","Installation of LSST stack for initial steps towards further understanding of the software components of the DM architecture. ",1 DM-4288,"L1 Function and Design Mtgs","2 days of design discussions and refining functional diagrams of the Alert Productions and Image Ingest system.",4 DM-4289,"Vendor Discussions re: specification documents","Specification document sent. Discussions with vendors covering document and schedule.",1 DM-4290,"SC15 Scheduling and Prep","Vendor appt scheduling and conference workshop scheduling, logisitics, etc for SuperComputing 15 in Austin",1 DM-4291,"LSST IaM Project",NULL,2 DM-4294,"investigate distributing automated lsst_apps builds via docker containers",NULL,15 DM-4295,"Run and document multinode integration tests on Openstack+Docker","Boot openstack machines using vagrant, then deploys docker images and finally launch multinodes tests. FYI, lack of DNS on OpenStack Cloud cause problems, but a vagrant plugin seems to solve this.",8 DM-4296,"Write up some introductory guides for Nebula usage","We write up some introductory guides for Nebula usage to enable new users to get started. These are located on Confluence under : https://confluence.lsstcorp.org/display/LDMDG/NCSA+Nebula+OpenStack+User+Guide",4 DM-4298,"want equiv of m1.xlarge flavor with smaller disk","I'd like to be able to build images with vcpus & ram from the {{m1.xlarge}} flavor that can be run on a {{m1.medium}} with it's smaller disk image, this would require a new flavor with 16GiB ram/8vcpus but only 40GiB of disk. Something along the lines of: {{openstack flavor create --ram 16384 --disk 40 --vcpus 8 ...}} Is that possible?",2 DM-4299,"Vagrant for Nebula OpenStack","Create and document a Vagrant configuration to use [~jhoblitt] lsstsw machine images on NCSA's Nebula OpenStack cloud.",4 DM-4301,"Convert banner and menu to react/flux","Add flux data model to capture menu and banner information. Convert banner and menu UI from gwt to react. As part of this task, bring in Fetch API to simplify client/server interactions.",8 DM-4302,"update obs_lsstSim","obs_lsstSim has seen some bitrot. In particular, the ingest task and the addition of the getExposureId methods on processImageTask have not been propagated to obs_lsstSim. This ticket will deal with those issues.",4 DM-4303,"re-deploy lsstsw on Jenkins","Pandas was added to the bin/deploy script in lsstsw to support sims development. This has already been merged to master in 4b1d1a0fa. The ticket is to ask that lsstsw be redeployed so the sims team can build branches that use pandas.",3 DM-4304,"Add unit testing into gradle build for Firefly's server-side code","Add a test task to Firefly's common build script. This can be used by any sub-project to run unit test. Added unit testing to Jenkins continuous integration job to ensure new code does not break unit testing.",1 DM-4306,"FY17 Implement Image Caching","When a file is requested, a cache maintained by the service is checked. If the file exists in the cache, it is returned. If the file does not exist, configurable rules are consulted to remove one or more files to make room for it in the cache, if necessary. (If no room is currently available because all cached files are being used, the request is blocked.) The file is then regenerated by invoking application pipeline code based on provenance and metadata information stored in the repository. The regenerated file is placed in the cache. This is documented in [LDM-152 §4.1|http://ldm-152.readthedocs.org/en/master/#image-file-services-baseline].",60 DM-4307,"Please add HSC tests to CI","In DM-3663 we (= [~price]) provided an integration test for processing HSC data through the stack with the intention that it should be integrated with the CI system. Having this test available and regularly run would be enormously helpful with the HSC port -- we've already run into problems which it could have helped us avoid (DM-4305).",1 DM-4309,"Try out nebula for stack developing and data processing","Follow instructions on: https://confluence.lsstcorp.org/display/LDMDG/NCSA+Nebula+OpenStack+User+Guide https://community.lsst.org/t/creating-a-nebula-instance-a-recipe/353 and create a nebula instance with the stack, update and install more packages in the stack, test by constructing a data repository and processing ISR with DECam raw images. ",2 DM-4310,"Missing Doxygen documentation","As of [2015-11-10 02:53.26|https://lsst-web.ncsa.illinois.edu/doxygen/xlink_master_2015_11_10_02.53.27/] there were 19 ""mainpages in subpackages"" available through Doxygen. In the next build, [2015-11-10 21:16.19|https://lsst-web.ncsa.illinois.edu/doxygen/xlink_master_2015_11_10_21.16.19/], most of them have vanished and we only provide links for {{ndarray}} and {{lsst::skymap}}. As of filing this issue, they were still missing from the [latest build|https://lsst-web.ncsa.illinois.edu/doxygen/x_masterDoxyDoc/]. Please bring them back!",1 DM-4311,"Oct. on-going support to Camera team in UIUC","Attend UIUC weekly meeting and give support as needed. ",2 DM-4312,"Nov. on-going support to Camera team in UIUC","Attend UIUC weekly meeting and give support as needed. ",2 DM-4313,"Configure sshd with pam_krb5 and document","Configure sshd in our test IAM VM to use pam_krb5 so the user gets a Kerberos ticket when logging in as discussed in our design doc. Document the configuration steps.",4 DM-4314,"Configure DAX web app with Kerberos authentication in IAM test VM","Install the DAX server (https://github.com/lsst?query=dax) in our test IAM VM at NCSA and configure it to use Kerberos authentication using standard Python flask Kerberos support. Document the process.",17 DM-4315,"Set up test IAM MariaDB instance with Kerberos","Install MariaDB in our test IAM VM at NCSA and configure it for Kerberos password authentication. I understand that Kerberos ticket authentication is not yet supported by MariaDB. Document the process and include references to the ongoing work to add Kerberos ticket support. Create example tables that demonstrate access control for different logged in users.",2 DM-4316,"Configure sssd with NCSA LDAP for accounts in test IAM VM","Configure [sssd|https://fedorahosted.org/sssd/] with NCSA LDAP for accounts in test IAM VM using instructions from Doug Fein.",5 DM-4317,"Configure httpd with SSL and mod_krb5 in IAM test VM","Get SSL certificate and configure httpd for mod_auth_krb5 authentication in IAM test VM. Document the setup.",4 DM-4323,"Replace fitsthumb in obs_subaru (port HSC-1196)","{{fitsthumb}} is now obsolete; all the functionality we need is available in {{afw}}. Further, we want to drop it as a dependency to make the job of integrating {{obs_subaru}} with CI easier.",1 DM-4324,"Bootstrap for new Sphinx/reST/RTD technical reports","Create a template repository for LSST (DM/SQuaRE) technical reports that are written in reStructuredText, built with Sphinx and published with RTD",2 DM-4325,"investigate distributing automated lsst_apps builds","(aggregation ticket for [previous existing]] tightly related tasks in progress related to binary distribution)",38 DM-4329,"Coadd_utils tests should run and skip if afwdata is missing","Currently, the {{coadd_utils}} tests are completely skipped at the scons layer if afwdata can not be located. This is bad for two reasons: 1. Are there any tests that can be run even if afwdata is missing?. 2. When we switch to a proper test harness (e.g. DM-3901) an important metric is the number of tests executed compared with the number of tests skipped. Each test file (or even each test) should determine itself whether it should be skipped based on afwdata availability. This should not be a global switch.",1 DM-4331,"ActiveMQ Broker upgrade","Version 5.8.0 of the ActiveMQ broker, which is not part of the regular distribution, is quite out of date. [~tjenness] is working upgrading the ActiveMQCPP library in DM-4330 to the current version, and this would be a good time to upgrade the broker to the latest release so we try and stay in sync.",1 DM-4332,"Sizing model fixes","* Improve API with LDM-144: need to separate spec of czar nodes from spec of worker nodes * Revisit model for qserv czar nodes * Fix chunk count for DiaObject (dbL2 F139) * Change low volume query result size, 0.5 GB --> 1 MB (sciReq G271). * Revisit model for shared scans and multiple releases * Add model for recovery from node failure for Query Access cluster * Consider modeling intermediate results (concluded that intermediate results are relatively small, and given they are distributed across the cluster, it is not worth modelling them) * Fix issue with transfer time of nightly products from Archive to Base --- * Add Dia* to scans * Consider modeling nodes for data loading * LDM-139 and LDM-141 don't document where MOPS Scratch comes from or what it's supposed to hold. * revisit bulk data transfer from HQ to Base (shipping disks --> over the network)",10 DM-4335,"The astrometry task should print a warning or throw an exception when there is no reference star in the field","When there is no reference star in the field the exception raised by anetAstrometry is identical to the one issued when the fit has not converged. astrometry (matchOptimisticB) is throwing an exception with the message lsst::pex::exceptions::InvalidParameterError: 'posRefBegInd too big'. I suggest a test to detect that there is not enough stars to fit the astrometry and to throw an exception accordingly.",1 DM-4336,"Global Metadata","Revisit design of the global metadata for LSST image and file archive. This includes understanding interactions between butler, local per repository metadata, global metadata, and metaserv. Sort out the dividing line between global metadata and dax (e.g., the schema https://github.com/lsst/dax_metaserv/tree/master/sql should live in cat/sql, not in dax) Deliverable: document describing the architecture of the global metadata, including how pipelines, users and SUI will interact with it. It should cover both writing/ingest and reading aspects.",60 DM-4337,"S17 Implement Image and File Archive","Implement fully working system capable of managing images and files. It will be used by the Alert Production system that we will be putting together in FY17",90 DM-4338,"F17 Integrate Image and File Archive with Alert Production",NULL,60 DM-4339,"FY18 Integrate Image and File Archive with DRP",NULL,79 DM-4340,"S17 Butler",NULL,100 DM-4342,"Disentangle log messages from different processes","It's difficult to disentangle interleaved log messages when running with multiprocessing. Two possible ways to do this would be: 1. Have each process write to a different log file. Maybe we could allow the user to specify log filenames including {{%(pid)s}} and {{%(hostname)s}}. More useful would be to allow the full range of keys from a {{dataId}}, but that might require some changes in how processing runs. 2. Prepend each line of the log with context information ({{dataId}}, pid, hostname), allowing the user to use {{grep}} to do the disentangling. To avoid overwhelming the log with the context information, a hash of the context information could be used instead, with the lookups published in the log before the processing starts. h5. Brief summary on the changes: - Chain logs of sub-tasks to their parent task logs - Add {{PrependedFormatter}} as the new default formatter for log files (specified through {{--logdest}} command line argument for {{cmdLineTask}}). The standard output to the terminal remains the same as before. - Any string can be used to label a pex_logging Log, and with {{PrependedFormatter}} this label prepends each log message. For {{cmdLineTask}}, the dataId is prepended. - The HSC's commit on prepending the timestamps is also ported to {{PrependedFormatter}}. - An example log message: {code:java} 2016-03-08T22:29:56.889933: [{'filter': 'r', 'tract': 0, 'patch': '2,3'}]: mergeCoaddDetections: Culled 1731 of 7751 peaks {code} ",12 DM-4343,"Add args to s3s3bucket CLI","Add {{source-bucket}} and {{dest-bucket}} arguments to {{s3s3bucket}} the command line script. This is to allow for one off duplication of buckets. Increment version to 0.1.10.",2 DM-4346,"tested upgraded activemqcpp package","The activemqcpp package was upgraded as part of DM-4330, and I tested it to be sure the upgrade was backwards compatible with the code that exercises it in ctrl_events. It is.",1 DM-4347,"dax_imgserv 2015_10.0 build error","{{2015_10.0}} has a build error under a current {{lsstsw/bin/deploy}} environment. Current speculation is that this is related to the conda version of numpy being upgraded to {{1.10.1}}.",1 DM-4348,"Replace XML-RPC with in-process communication","With all recent development in CSS sector we should be able now to get rid of Python in czar entirely. This is also a good opportunity to move from XML-RPC between proxy and czar with direct in-process ""communication"". Daniel said it's a good idea :) ",10 DM-4349,"Fix publishing script async issue and add additional release notes.","Async command execution causes unpredictable and unreliable results. Switches to synchronous where possible. Also add additional description to the release notes.",2 DM-4351,"Write technote on the new technical note platform","Write a technote about the platform that github.com/lsst-sqre/lsst-technote-bootstrap lets DM members publish in. Discuss current status and outline future plans.",6 DM-4352,"Design Mtg, Review and discussions of L1 processing","Design/planning meeting for L1 system. Materials read and discourse discussions on EFD, MOPs, T&S docs and calibration production",3 DM-4353,"Reviewing quotes, power requirements, rack layouts","Review of multiple quotes from multiple vendors across full year of purchases. Derived power requirements and rack layouts for placement, networking, electrical work discussions to begin.",3 DM-4354,"SC15 Scheduling, Processor Futures Refresh","Scheduling of several more meeting opportunities for next week. Also, time spent reviewing NDA materials on processor futures. ",1 DM-4355,"S19 Run Large Scale Tests",NULL,26 DM-4356,"FY19 Implement Qserv Software Upgrading Tools",NULL,53 DM-4357,"FY19 Finalize Internal DRP Database","Final changes to make DRP database commissioning ready.",79 DM-4360,"obs_subaru fails to compile after DM-3200","Due to atypical calls in {{obs_subaru}}'s {{hsc/SConscript}} to run scripts in the {{bin}} directory, {{obs_subaru}} fails to compile after the changes made in DM-3200.",1 DM-4362,"SuperTask phase 1 implementation","This story represents the implementation of the first part of the SuperTask framework design,",8 DM-4363,"Implement configuration for activator parsing","Need to add configuration to the CmdLineActivator in new Workflow tasks",2 DM-4364,"First implementation demo","First stage demo of Super Task and WorkFlowTask Framework",4 DM-4365,"write documentation for registry free repo",NULL,1 DM-4366,"Improve overscan correction for DECam raw data","Currently, the default overscan correction from IsrTask is used for processing DECam raw data. Overscan subtraction is done one amplifier at a time. However, a bias jump occurs due to the simultaneous readout of the smaller ancillary CCDs on DECam, some images show discontinuity in the y direction across one amplifier, as in the example screen shot. This ticket is to improve overscan correction for DECam data so to mitigate this discontinuity in the ISR processing. Arrangement of CCDs on DECam: http://www.ctio.noao.edu/noao/sites/default/files/DECam/DECamPixelOrientation.png h3. More details: There are 6 backplanes in the readout system, shown by the colors in DECamPixelOrientation.png. In raw data files, the CCD's backplane is noted in the header keyword ""FPA"". Examination of some images suggests that science CCDs on orange and yellow backplanes show bias jump at 2098 pixels from the y readout. That is the y size of the focus CCDs. h3. Actions: For CCDs on the affected backplanes, divide the array into two pieces at the jump location, and do overscan correction on the upper and lower pieces separately. ",5 DM-4367,"Fix bug and add unit tests for PsfShapeletApprox ","We discovered during this Sprint that this plugin was giving us faulty values for all the models except for SingleGaussian. I will fix that bug on this issue. Obviously, a better unit test would have caught this. I am adding a DoubleGaussian unit test, plus a test that the default models provide different results. Also a timing test for all the models, as we do not really have enough information about the performance of the shapelet approximation. ",3 DM-4368,"Duration for various ShapeletPsfApprox Models","This is just a report of the amount of time it takes to run ShapeletPsfApprox and CModel over 10000 galaxies from GalSim",2 DM-4369,"Migrate lsst/ci_hsc repo to git-lfs.lsst.codes","The github lfs backed repo https://github.com/lsst/ci_hsc needs to be migrated to git-lfs.lsst.codes. A sanity check for any other ""live"" lfs repos under the lsst github org might also be a good idea.",4 DM-4370,"Migrate testdata for DECam from disk to git-lfs","There is a package full with test data for the obs_decam. It is an eups package currently, but not a git repository. I would like to migrate that into our hosted git-lfs so the obs_decam package can be built by Jenkins. [~jmatt] I'm hoping you would be willing to handle this for me. If not I can find somebody else. Thanks!",1 DM-4373,"HSC backport: Add tract conveniences","This is a port of [HSC-715: Add tract conveniences|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-715]. Here is the original description: {panel} In regular HSC survey processing, we'll run with a ""rings"" skymap to cover the entire survey area. meas_mosaic does not currently efficiently or conveniently iterate over tracts. For example: {code} mosaic.py /tigress/HSC/HSC --rerun price/cosmos --id field=SSP_UDEEP_COSMOS filter=HSC-R {code} Note the lack of a tract in the --id specifier — we want to iterate over all tracts. This is not currently possible. Instead, if we do not know the tract of interest (which the user should not be required to know), we have to iterate over all the tracts (e.g., tract=0..12345), but the user should not be required to know the number of tracts, and this is slow (and possibly memory-hungry: currently consuming 11GB on tiger3 just for 12 exposures). We need an efficient mechanism to iterate over all tracts by not specifying any tract on the command-line. {panel} As this functionality was added specifically for {{meas_mosaic}}, it was going to be ported as part of DM-2674. Due to a recent desire to use this functionality, this ticket will be ported here.",1 DM-4375,"A slimmer testdata_decam","Before this ticket, the files in {{testdata_decam}} are as they are downloaded from the archive. Some are MEF, and the total is a bit big (1.2G). I made a trimmed down version of {{testdata_decam}}, 109M and available here: [https://github.com/hchiang2/testdata_decam.git] I trimmed it down by only saving the primary HDU and one data HDU. However the unit tests in {{getRaw.py}} become less meaningful and I am not sure if we really want to do this, because some complexities of DECam files are about the MEF. {{getRaw.py}} tests Butler retrieval of multiple dataset types, in particular tests if the correct HDU is retrieved. Nonetheless, {{getRaw.py}} can pass (with branch u/hfc/DM-4375 of obs_decam) Note: the old testdata_decam still live on lsst-dev:/lsst8/testdata_decam/",2 DM-4376,"Learn and setup nebula as a development machine","Personally establish Nebula as a stack development platform.",2 DM-4377,"Review VAO/IVOA protocols for use in LSST IAM","Following the good principals of re-using prior work, review VAO/IVOA protocols for use in LSST IAM, in particular the [IVOA Credential Delegation Protocol|http://www.ivoa.net/documents/latest/CredentialDelegation.html] and include a summary in our LSST IAM Design Doc.",1 DM-4379,"X16 Revisit Public Interfaces / ADQL","We need to revisit the interfaces we will be exposing to public, wiht particular focus on ADQL vs SQL92 (mysql flavor?). Deliverable: recommendation which public interfaces should be exposed to users from the Data Access Services, with particular focus on ADQL vs mysql-flavor.",11 DM-4381,"""SHUTOFF"" nebula instances consume core/ramIt quota","It appears that halted/shutoff instances have no effect on resource quota usage. Eg: {code:java} $ openstack server list +--------------------------------------+-----------------------+-------------------+----------------------------------------+ | ID | Name | Status | Networks | +--------------------------------------+-----------------------+-------------------+----------------------------------------+ ... | 1956c6d0-8aec-4f42-a781-8a68fd10179d | el7-jhoblitt | SHUTOFF | LSST-net=172.16.1.171, 141.142.208.150 | ... $ nova absolute-limits +--------------------+--------+--------+ | Name | Used | Max | +--------------------+--------+--------+ | Cores | 141 | 150 | | FloatingIps | 0 | 10 | | ImageMeta | - | 128 | | Instances | 44 | 100 | | Keypairs | - | 100 | | Personality | - | 5 | | Personality Size | - | 10240 | | RAM | 342016 | 400000 | | SecurityGroupRules | - | 20 | | SecurityGroups | 1 | 10 | | Server Meta | - | 128 | | ServerGroupMembers | - | 10 | | ServerGroups | 0 | 10 | +--------------------+--------+--------+ $ openstack server delete 1956c6d0-8aec-4f42-a781-8a68fd10179d $ nova absolute-limits +--------------------+--------+--------+ | Name | Used | Max | +--------------------+--------+--------+ | Cores | 133 | 150 | | FloatingIps | 0 | 10 | | ImageMeta | - | 128 | | Instances | 43 | 100 | | Keypairs | - | 100 | | Personality | - | 5 | | Personality Size | - | 10240 | | RAM | 325632 | 400000 | | SecurityGroupRules | - | 20 | | SecurityGroups | 1 | 10 | | Server Meta | - | 128 | | ServerGroupMembers | - | 10 | | ServerGroups | 0 | 10 | +--------------------+--------+--------+ {code} ",2 DM-4382,"Port registryInfo.py from obs_subaru into Butler","in butler (probably butlerUtils), users would like the ability to dump info from a repository's sqlite registry to text (console output). This is already implemented in {{obs_subaru/.../registryInfo.py}}, and basically just needs to be ported to butler in a sensible place. There are a few cases that assume certain columns are present and we need either to make the script more generic, or [~rhl] suggests that maybe we need to standardize the registries.",8 DM-4383,"Avoid restarting czar when empty chunk list changes","Currently czar caches empty chunk list after it reads the list from file. This complicates things when we need to update the list, integration test for example has to restart czar process after it loads new data to make sure that czar updates its cached list. Would be nice to have simpler mechanism to resetting cached list in czar without restarting it completely. It could be done via special query (abusing FLUSH for example) or via sending signal (problematic if czar runs remotely). This can be potentially useful even after we replace empty chunk list file with some other mechanism as I expect that cache will stay around even for that.",2 DM-4386,"Clean up ProcessCcdDecam","ProcessCcdDecam needs some cleanup: * {{run}} method simply delegates to the base class * {{propagateCalibFlags}} is a no-op (deliberately in {{cab69086}}, need to explore if the original problem still exists) * The config overrides (in config/processCcdDecam.py): ** Uses the catalog star selector, which isn't wise given the current heterogeneity of reference catalogs. ** Sets the background {{undersampleStype}} to {{REDUCE_INTERP_ORDER}}, which is the default.",1 DM-4387,"Skymap fails tests on testFindTractPatchList","When skymap is built and healpy is loaded, {{testFindTractPatchList}} fails with: {quote}====================================================================== FAIL: Test findTractPatchList ---------------------------------------------------------------------- Traceback (most recent call last): File ""/Users/ctslater/lsstsw/build/skymap/tests/SkyMapTestCase.py"", line 245, in testFindTractPatchList self.assertClosestTractPatchList(skyMap, [tractInfo.getCtrCoord()], tractId) File ""/Users/ctslater/lsstsw/build/skymap/tests/SkyMapTestCase.py"", line 284, in assertClosestTractPatchList tractPatchList = skyMap.findClosestTractPatchList(coordList) File ""/Users/ctslater/lsstsw/build/skymap/python/lsst/skymap/baseSkyMap.py"", line 146, in findClosestTractPatchList tractInfo = self.findTract(coord) File ""/Users/ctslater/lsstsw/build/skymap/python/lsst/skymap/healpixSkyMap.py"", line 97, in findTract index = healpy.ang2pix(self._nside, theta, phi, nest=self.config.nest) File ""/Users/ctslater/lsstsw/stack/DarwinX86/healpy/1.8.1+12/lib/python/healpy-1.8.1-py2.7-macosx-10.5-x86_64.egg/healpy/pixelfunc.py"", line 367, in ang2pix check_theta_valid(theta) File ""/Users/ctslater/lsstsw/stack/DarwinX86/healpy/1.8.1+12/lib/python/healpy-1.8.1-py2.7-macosx-10.5-x86_64.egg/healpy/pixelfunc.py"", line 110, in check_theta_valid assert (np.asarray(theta) >= 0).all() & (np.asarray(theta) <= np.pi + 1e-5).all(), ""Theta is defined between 0 and pi"" AssertionError: Theta is defined between 0 and pi{quote} This was missed during regular CI testing since healpy is not normally setup. ",1 DM-4391,"Update testCoadds.py to accommodate changes in DM-2915","As of DM-2915, the config setting: {code}self.measurement.plugins['base_PixelFlags'].masksFpAnywhere = ['CLIPPED']{code} is set as a default for {{MeasureMergedCoaddSourcesTask}}. However, this *CLIPPED* mask plane only exists if a given coadd was created using the newly implemented {{SafeClipAssembleCoaddTask}}. If a coadd was built using {{AssembleCoaddTask}}, the *CLIPPED* mask plane is not present, so the above default must be overridden to exclude it when using {{MeasureMergedCoaddSourcesTask}}. This is the case for the mock coadd that is assembled in the unittest code in {{testCoadds.py}}, so the config needs to be set for the test to run properly. Note that the associated tests for {{SafeClipAssembleCoaddTask}} will be added as part of DM-4209.",1 DM-4393,"Get analysis script working for HSC/LSST stack comparisons","A script for performing pipeline output QA is under development for HSC. The script provides many useful tools for plotting and analyzing pipeline outputs on single visits and coadds. This is of general use for LSST and, in particular, will be adapted/expanded to include tools for the direct comparisons of identical data sets processed with both the HSC and LSST pipelines (i.e. DM-2984). Appropriate adaptations for this script to run with the LSST stack will be made here (with the understanding that development is still ongoing on HSC and further adaptations will be accommodated as necessary/desired). See [HSC-1320|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1320] and [HSC-1359|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1359] for details and examples of the output of this script.",6 DM-4395,"Update cmsd configuration for multi-node tests","A particular cmsd configuration parameter prefixes a hardcoded path for QueryResource, which needs to be removed. This seems to appear only during multi-node tests.",1 DM-4396,"ctrl_execute test fails to find test binary","There's a test in ctrl_execute that exercises the bin/dagIdInfo.py test program. Since the rewrite_shebang rewrites happen after the tests are executed, the test that looks for the bin/dagIdInfo.py binary fails, since it's not there before the tests execute.",1 DM-4397,"Scale CommandLineTask multiprocessing timeout with workload","The default timeout value for aborting a multiprocessing run in CommandLineTask is too short. Currently if no time length is supplied by the user, the default value gets set to 9999s. However if a processing task is quite large it is possible for the processing pool to take much longer to arrive at the result. Currently if the processing pool does not complete it's run within that time limit, python multiprocessing will throw a timeout error. The timeout value should be scaled such that the supplied value is assumed to be the timeout length for one processing task, and should be scaled by the number of tasks divided by the number of cpus available. The command line task documentation should be updated to reflect this change.",2 DM-4398,"Fix regexp for gcc48","DM-2622 inttoduced some regexes which raise exceptions when built with gcc48 (e.g. on centos7). gcc48 support for regexes is generally broken, so it's better to replace that with boost regexes.",1 DM-4399,"ctrl_execute test fails under El Capitan","The test/testDagIdInfo.py because it runs a script from bin.src, rather than bin. This test needs to be rewritten.",1 DM-4400,"SuperTask demo on other older tasks","The exampleCmdLine task worked fine, need to show demo for other tasks from pipe_tasks",4 DM-4401,"W16 Qserv Refactoring #2","Additional refactoring of Qserv as found necessary in W16",70 DM-4402,"Experiment with light-weight SQL databases for secondary index","Evaluate the use of light-weight SQL, such as InnoDB, TokuDB (now Kyoto Cabinet), or RocksDB to create and manage the secondary index.",8 DM-4406,"Review of [DM-2983] part 2","Second part and final of reviewing DM-2983",2 DM-4407,"Debug Qserv on ccqserv125..ccqserv149","It seems that some chunkQuery doesn't return on long queries like ""Select count(*) From Object"" The query stall and print in czar log: {code} 2015-11-24T15:07:55.324Z [0x7f06b37fe700] INFO root (core/modules/qdisp/Executive.cc:395) - Still 9 in flight. {code} If we look in the logs with next commands: {code:bash} cat /qserv/run/var/log/qserv-czar.log | egrep ""Executive::add\(job\(id="" | grep LSST | cut -d'=' -f2 | cut -d' ' -f1 | sort > JOBID_ADD.txt cat /qserv/run/var/log/qserv-czar.log | egrep ""markCompleted"" | cut -d'(' -f 3 | cut -d',' -f1 | sort > JOBID_DONE.txt {code} And then {code} qserv@ccqserv125:/qserv$ diff JOBID_ADD.txt JOBID_DONE.txt 1826d1825 < 2639 2217d2215 < 2991 2904d2901 < 3609 3088d3084 < 3775 3152d3147 < 3832 3433d3427 < 4085 4227d4220 < 480 5478d5470 < 5926 6937d6928 < 7239 {code} 9 jobs aren't completed on czar. If we look the the chunk_id of one of this job: {code} /opt/shmux/bin/shmux -c ""locate 5299"" ccqserv{125..149}.in2p3.fr ... ccqserv148.in2p3.fr: /qserv/data/mysql/LSST/Object_5299.MYD ccqserv148.in2p3.fr: /qserv/data/mysql/LSST/Object_5299.MYI ccqserv148.in2p3.fr: /qserv/data/mysql/LSST/Object_5299.frm ... {code} Data is on worker node and onccqserv148 xrootd log: {code} [2015-11-24T15:07:42.990Z] [0x7ffbc8244700] INFO root (core/modules/xrdsvc/SsiSession.cc:125) - Enqueued TaskMsg for Resource(/chk/LSST/5299) in 0.000465 {code} But a thread seems to be locked: {code} #gdb on ccqserv148 xrootd process: (gdb) thread 7 [Switching to thread 7 (Thread 0x7ffbad7fa700 (LWP 192))] #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 185 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory. (gdb) bt #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffbd4c45c7c in std::condition_variable::wait(std::unique_lock&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #2 0x00007ffbccbe8db9 in std::condition_variable::wait >(std::unique_lock &, lsst::qserv::wsched::BlendScheduler::) ( this=0xfba620, __lock=..., __p=...) at /usr/include/c++/4.9/condition_variable:98 #3 0x00007ffbccbe88f9 in lsst::qserv::wsched::BlendScheduler::getCmd (this=0xfba5c0, wait=true) at core/modules/wsched/BlendScheduler.cc:156 #4 0x00007ffbccba3b07 in lsst::qserv::util::EventThread::handleCmds (this=0xfbe890) at core/modules/util/EventThread.cc:45 #5 0x00007ffbccbab137 in std::_Mem_fn::operator()<, void>(lsst::qserv::util::EventThread*) const (this=0xfbe900, __object=0xfbe890) at /usr/include/c++/4.9/functional:569 #6 0x00007ffbccbaafff in std::_Bind_simple (lsst::qserv::util::EventThread*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0xfbe8f8) at /usr/include/c++/4.9/functional:1700 #7 0x00007ffbccbaae45 in std::_Bind_simple (lsst::qserv::util::EventThread*)>::operator()() (this=0xfbe8f8) at /usr/include/c++/4.9/functional:1688 #8 0x00007ffbccbaacfa in std::thread::_Impl (lsst::qserv::util::EventThread*)> >::_M_run() (this=0xfbe8e0) at /usr/include/c++/4.9/thread:115 #9 0x00007ffbd4c49970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x00007ffbd50ae0a4 in start_thread (arg=0x7ffbad7fa700) at pthread_create.c:309 #11 0x00007ffbd43b904d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 {code} Something seems to prevent return of chunk query result... ",5 DM-4408,"HSC backport: fix memory leak in afw:geom:polygon","This is a backport of a bug fix that got included as part of [HSC-1311|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1311]. It is not related to that issue in particular, so is being ported here as an isolated bug fix. {panel} Original commit message: pprice@tiger-sumire:/tigress/pprice/hsc-1311/afw (tickets/HSC-1311=) $ git sub commit 55ad42d37fd1346f8ebc11e4077366dff4eaa87b Author: Paul Price Date: Wed Oct 21 10:59:56 2015 -0400 imageLib: import polygonLib to prevent memory leak When doing ""exposure.getInfo().getValidPolygon()"", was getting: swig/python detected a memory leak of type 'boost::shared_ptr< lsst::afw::geom::polygon::Polygon > *', no destructor found. This was due to the polygonLib not being imported in imageLib. Using polygonLib in imageLib then requires adding polygon.h to all the swig interface files that use imageLib.i. examples/testSpatialCellLib.i | 1 + python/lsst/afw/cameraGeom/cameraGeomLib.i | 1 + python/lsst/afw/detection/detectionLib.i | 1 + python/lsst/afw/display/displayLib.i | 1 + python/lsst/afw/geom/polygon/Polygon.i | 1 + python/lsst/afw/image/imageLib.i | 2 ++ python/lsst/afw/math/detail/detailLib.i | 1 + python/lsst/afw/math/mathLib.i | 1 + 8 files changed, 9 insertions(+) {panel}",1 DM-4410,"Port detection task footprint growth changes from HSC","In hsc the default behavior for the detection task is to updated footprints with a footprint which has been grown by the psf. This behavior needs to be ported to LSST, as some source records have footprints which are too small. When making this change, the new default needs to be overridden for the calibrateTask, as it needs the original size. The port includes 8e9fb159a3227f848e0db1ecacf7819599f1c03b from meas_algorithms and 8bf0f4a44c924259d9eefbd109aadec7d839e0f2 from pipe_tasks",5 DM-4412,"Write a DM Collaborative Workflow document","Document our procedures for collaborative development with JIRA, Git and GitHub for the new docs.",11 DM-4414,"Add git-lfs to packer-newinstall","git-lfs is not available in our deliverables. Artifacts (binaries) such as VM images and docker data containers should provide a stable version of git-lfs.",2 DM-4415,"Meetings - Nov 2015","Verification dataset meetings, Illinois DES meeting, single-frame processing discussions, supertask meeting, OpenStack User meeting",2 DM-4416,"Other LOE - Nov 2015","Local LSST group meetings, NCSA postdoc meetings, NCSA Physics & Astronomy theme meeting, or other local meetings, events, or tasks to comply with NCSA policies",5 DM-4417,"Crash course on using git-lfs","Learn to install and use git-lfs; help testing with migrating {{testdata_decam}} to git-lfs; verify tests pass with the new repository (DM-4370).",1 DM-4418,"Learn about the task design in ISR processing","Learn the concept behind the previous API changes (RFC-26) in the tasks of ISR processing, and data storage/retrieval involved. ",3 DM-4419,"Explore basic middleware and orchestration tools","Use {{runOrca}} to launch jobs through {{lsst-dev}} and do some single frame processing with it. Also learn a little about process execution. ",3 DM-4420,"Avoid bash usage in batch submission","{{ctrl_pool}} currently creates bash submission scripts with an explicit {{/bin/bash}} bang line. [~rhl] [argues| https://github.com/lsst/ctrl_pool/commit/047f0de5a682ad9e9a6f65ccc7cc296e0a0d4ee7#commitcomment-14573759] on the review of DM-2983 that we should using posix shell constructions instead.",1 DM-4421,"faulty assumption about order dependency in ctrl_event unit tests","A recent change to daf_base uncovered a couple of faulty tests in ctrl_events that incorrectly assumes the order in which assumed the order in which data in a PropertySet would be received. We can't assume which order these values will be put into the property set, and therefore into the list retrieved from the Event object.",1 DM-4422,"Image preparation time at server side measurement ","Setup the mechanism to measure the time needed to prepare the image (generate the image in PNG or other suitable format) for client display. ",20 DM-4423,"Image preparation time at server side measurement","Measure the time needed to prepare the image on server side for client display. Reach the goal of less than one second.",20 DM-4424,"Image preparation time at server side measurement","Measure the time needed to prepare the image on server side for client display. Reach the goal of less than half second.",20 DM-4425,"setup the measurement for Image rendering time to display at web UI","Setup the mechanism to measure the time needed to render the image after the data received by the client for display. ",30 DM-4426,"measure Image rendering time to display at web UI","Measure the image rendering time, reach the goal of 1 second.",30 DM-4427,"measure Image rendering time to display at web UI","Measure the image rendering time, reach the goal of 0.5 second. ",30 DM-4428,"Remove Task.display()","As of DM-927 (included in release 9.2, end of S14), {{lsst.pipe.base.task.Task.display()}} was marked as deprecated. It should now be removed.",3 DM-4429,"Revisit mysql connections from czar","Need to revisit connections we maintain from czar to mysql. This include revisiting whether we need both sql/SqlConnection and mysql/MySqlConnection classes. (In some cases, like in InFileMerger we have instances of both, which gets very confusing.)",12 DM-4431,"setup mechanism to measure the query response time ","Setup the method to measure the response time after query has been submitted from client. 1. query sent to the data provider from client 2. result returns from data provider 3. result displayed in the client We can measure 1. the time from query submission to been sent to data provider 2. the time from data returned from provider to been displayed in the client",30 DM-4432,"measure the query response time ","Measure the time from query returned from data provider to been displayed in the client.",30 DM-4433," measure the query response time ","Measure the time from query returned from data provider to been displayed in the client.",30 DM-4434,"Setup the load test for Web UI portal ","Setup the load test system to measure the performance of web UI portal",50 DM-4435,"load test of Web UI portal: support 100 concurrent users","Run the load test system with 100 concurrent users. Measure the performance against other KPM epics in the same cycle. ",50 DM-4436,"load test of Web UI portal: support 100 concurrent users","Run the load test system with 100 concurrent users. Measure the performance against other KPM epics in the same cycle. ",50 DM-4437,"Improve docker storage backend on RedHat-like distributions","Solve startup log message RedHat-like distro: ""level=warning msg=""Usage of loopback devices is strongly discouraged for production""? This is due to use of DeviceMapper (default package option on RedHAt-like distros) without a dedicated hard-disk, use of ""overlay"" backed storage seems better. ",4 DM-4438,"Replace sed with stronger template engine in docker scripts","Dockerfile are generated using templates and sed, this should be strengthened.",2 DM-4439,"Remove useless xrootd client parameters","This extract of etc/sysconfig/docker: {code:bash} # used by qserv-czar export QSW_RESULTDIR=${QSERV_RUN_DIR}/qserv-run/tmp # Disabling buffering in python in order to enable ""real-time"" logging. export PYTHONUNBUFFERED=1 export XRD_LOGLEVEL=Debug export XRDSSIDEBUG=1 # Increase timeouts, without that, long-running queries fail. # The proper fix is coming, see DM-3433 export XRD_REQUESTTIMEOUT=64000 export XRD_STREAMTIMEOUT=64000 export XRD_DATASERVERTTL=64000 export XRD_TIMEOUTRESOLUTION=64000 {code} Has to be moved to: {code:bash} # used by qserv-czar export QSW_RESULTDIR=${QSERV_RUN_DIR}/qserv-run/tmp export XRD_LOGLEVEL=Debug export XRDSSIDEBUG=1 # Disabling buffering in python in order to enable ""real-time"" logging. export PYTHONUNBUFFERED=1 {code} And then tested in multi-node, and on in2p3 cluster.",1 DM-4440,"Remove QSW_RESULTPATH and XROOTD_RUN_DIR if useless","These parameters may be useless (see DM-4395). If yes they can be removed to simplify configuration procedure.",2 DM-4442,"IAM process for granting data access rights","Document a process for granting of data access rights to LSST users according to the Data Access White Paper ([Doc-53733|http://ls.st/Document-5373]). On the wiki: https://confluence.lsstcorp.org/display/LAAIM/Granting+Data+Access+Rights",1 DM-4443,"Please document the --rerun option","DM-3371 adds the {{--rerun}} option to command line tasks. The help for this option reads: {quote} rerun name: sets OUTPUT to ROOT/rerun/OUTPUT; optionally sets ROOT to ROOT/rerun/INPUT {quote} While essentially correct, that's not particularly helpful in understanding what's actually going on here. A motivation and description of this functionality is available in RFC-95: please ensure that, or some variation of it, is included in the stack documentation.",1 DM-4444,"ISR and calibration of a tiny set of DECam raw data","Learn more about the beginning steps of single frame processing by processing a small subset of DECam Stripe 82 raw data (~10 visits) and performing instrument signature removal with features currently implemented.",6 DM-4445,"IaM work in November","Work done in support of LSST's IaM efforts.",1 DM-4446,"Management for November","Centered around DMLT meeting, design process and hiring, in addition to general steering of activities at NCSA ",6 DM-4447,"November TOWG/opeartions design work","Towg attendance/ note + participating in Beth's group. Detailed WBS for DM, Condensed WBS to show high level, Effort estimates, point out problematic thinking in the estimates. ",8 DM-4448,JCC,"Two day JCC activity at NCSA, including extended JCC meeting with HEP centers likely to host people exploiting LSST Data. writeup of extended meeting is here: https://confluence.lsstcorp.org/display/JCC/Extended+JCC+meeting+--+2015-11-23 in the JCC section. organize, coordinate, and write up meeting notes. ",5 DM-4449,"Design refinement for the L1 system","Further specification of L1 design, Long list of Questions for group but handled by GDF, began procession replies. Understood Chilean Buffer in L1 system could be eliminated, posed question about systems engineering process needed to support this. Genera work casing further developing the design into WBS -- which is not 30+ lines of detail Began L1 con ops ,to guide group Learned of some (possible undocumented) ""fifth device"".",11 DM-4450,"Data Distrib proto (dec)",NULL,15 DM-4451,"S17 Implement Async Queries in Qserv","* Design and implement *basic* system for determining whether particular query is synchronous or asynchronous. The complete version will come through DM-1490. Note that this work is related to shared scans (e.g., we need to know what scans we have running) * Design SQL API for starting and interacting with async queries. * Modify Qserv to support async queries (starting, getting status, retrieving results) Note, async queries are indirectly related to authentication (users should not see each other' async queries). Deliverable: Qserv that accepts and executes queries asynchronously, and allows users to retrieve results.",50 DM-4452,"HTML5 Sphinx theme for technotes","Build a Sphinx theme for the Technote platform. Treat this work as exploratory, proof of concept work for customizing the HTML, CSS and JS of the software docs. Objectives are 1. Create a Sphinx theme repo 2. Show how gulp can be used to help develop web assets for the theme 3. Establish a pattern for table contents columns that scroll independently of the main article 4. Show how we can implement a HTML5 rst builder in documenteer to fix permalink issues and build true HTML5 output.",2 DM-4453,"Finish documentation and comments on SuperTask ","Need to finish documentation, implementation and comments on the code",4 DM-4454,"Fix multiple patch catalog sorting for forcedPhotCcd.py","{{forcedPhotCcd.py}} is currently broken due to the requirement of the {{lsst.afw.table.getChildren()}} function that the *SourceCatalog* is sorted by the parent key (i.e. {{lsst.afw.table.SourceTable.getParentKey()}}). This occurs naturally in the case of *SourceCatalogs* produced by the detection and deblending tasks, but it may not be true when concatenating multiple such catalogs. This is indeed the case for {{forcedPhotCcd.py}} as a given CCD can be overlapped by multiple patches, thus requiring a concatenation of the reference catalogs of all overlapping patches. There two places in the running of {{forcedPhotCcd.py}} where calls to {{getChildren()}} can cause a failure: one in the {{subset()}} function in {{references.py}}, and the other in the {{run}} function of *SingleFrameMeasurementTask* in {{sfm.py}}.",2 DM-4455,"Understand how the proposed interfaces fit with Qserv code","Understand how the interfaces proposed by [~abh] in DM-3755 fit with the existing qserv code.",3 DM-4456,"Re-locate LSST PS server and configure to reside on new layer2 circuits",NULL,2 DM-4457,"Investigate MemSQL","Take a look at the MemSQL distributed database.",8 DM-4458,"Week end 11/07/15","Support for lsst-dev cluster, OpenStack, and accounts for week ending November 7, 2015.",2 DM-4459,"Week end 11/14/15","Support for lsst-dev cluster, OpenStack, and accounts for week ending November 14, 2015.",4 DM-4460,"Week end 11/21/15","Support for lsst-dev cluster, OpenStack, and accounts for week ending November 21, 2015.",2 DM-4462,"New equipment setup and configuration (week end 11/07/15)","* Three Dell R730's: ** Mount in A row racks ** Complete Bios updates * Moved ~25 VMs over to new lsst-vsphere infrastructure * Setup lsst-condor\[1-6\] VMs * Setup lsst-esxmac1 with networking * Fixed networking issues on new lsst-esx1 server (an undocumented host was squatting on the IP address)",5 DM-4463,"New equipment setup and configuration (week end 11/14/15)","* Received Dell iDrac license upgrades for new Dell R730 servers * Received and configured VMware vSphere licenses from CDW-G & AURA * Converted three physical systems to VM's: ** lsst-nagios *** Problems with software mirror raids. **** VMware converter does not see software raided drives. **** Split the raid 1 into discrete drives. **** Chose sda to modify - failed as sad was faulty. *** Built new Centos 6.6 VM **** Used CrashPlan to rebuild. ** lsst7 - converted after learning how to convert system to a fixed IP subnet ** lsst8 - converted *** Debugged lsst8 system migration to vmware.  Partition table was invalid and was preventing move * Finished moving all VMs to new lsst-vsphere infrastructure",6 DM-4464,"New equipment setup, configuration, and regular monthly maintenance (week end 11/21/15)","* Virtualized physical system lsst-xfer * Worked with bmather or dell Dirac licensing issue * Cleaned up old and new hosts in NCSA DNS, Nagios monitoring, and Qualys vulnerability scanner * Completed connections for six UPS",2 DM-4465,"New equipment setup and configuration (week end 11/28/15)","* Obtained Dell iDRAC Enterprise licenses & upgraded 4 of the 13 servers * Installed base VM templates for OS X 10.8-10.11",1 DM-4466,"Decommissioning old equipment (week end 11/14/15)","* Shutdown 17 (all) old ESXi servers * Shutdown 3 old condor servers",1 DM-4467,"Decommissioning old equipment (week end 11/21/15)","* Shutdown last 3 old condor servers * Shutdown lsst-netem, lsst-ps, & lsst-ps-base servers * Surplussed equipment: ** NCSA servers ( 5 Dell 1950's, 2 Dell 2950) repurposed from A22 to C17 ** Moved blade chassis to C20 * Move lsst-test systems in A23",2 DM-4469,"Write DM Git LFS Guide","Refactor words from DM-4412 into a top-level Doc page for using Git LFS. This will leave DM-4412 more as a Collaboration Workflow document.",1 DM-4471,"Consulting in November","Review of design documents, correspondence with the design team regarding Data Center details and floor space, and conferences via web links.",4 DM-4472,"Shared scan implementation","Fine tune the API proposed in DM-3755 and implement it.",10 DM-4473,"Improvements to logging in xrootd","Improve logging in xrootd to make it more compatible with qserv logging.",6 DM-4474,"Fit Visualizer porting: Begin","Display fits image, server round trip, organize initial data structures, initial render",16 DM-4475,"FITS Visualizer porting: group, group scrolling",NULL,6 DM-4476,"FITS Visualizer porting: zooming: group zooming, zoom fit, zoom fill, active plot selection",NULL,6 DM-4477,"Upgrade to react 0.14.3",NULL,2 DM-4478,"Fit Visualizer porting: create toolbar",NULL,6 DM-4479,"FITS Visualizer porting: Add canvas drawing infrastructure",NULL,10 DM-4480,"Fit Visualizer porting: Distance Tools",NULL,4 DM-4481,"Fit Visualizer porting: Drawing Target Center ",NULL,3 DM-4482,"FITS Visualizer porting: Marker tool","improve the menu organization from the current one when doing the migration. The implementation includes: * DrawObj for marker * Marker dropdown list under the marker icon to show the marker and footprint items (will be added later). * Marker drawing layer rendering and operation including action creator and action dispatch functions. * Marker UI component shown in Drawing Layers popup. The implementation set up some work which can be similarly expanded to footprints in the future. * Title of marker layer on the layer control is like ""show: "". The title change as the text you enter for the label. * the mouse changes to a pointer once the cursor moves onto the marker, and the mouse becomes a resize sign when the cursor moves on the handler of the marker. * When the cursor becomes a pointer, the marker can be relocated by dragging the mouse, and when the cursor becomes a resize one, the marker can be resized by dragging the mouse. * the marker size changes as the image is zoomed. Maker drawing layer operation: * select 'Add Marker' from dropdown list to add a new drawing layer * click anywhere to locate the newly added marker * click and drag the mouse to relocate or resize the marker: for relocation: click and drag inside the circle, then drag the mouse. for resize: click inside the circle, then click and drag the handler to resize * label and its location are set from Marker tool UI ",10 DM-4483,"FITS Visualizer porting: Grid drawing",NULL,10 DM-4484,"FITS Visualizer porting: Catalog drawing",NULL,6 DM-4485,"FITS Visualizer porting: Region Drawing","region drawing for the following regions, circle, ellipse, box, polygon, point (circle, box, diamond, cross, x, arrow, boxcircle), line, text, annulus, box annulus, ellipse annulus. (annulus is made for the case with at least two repeated regions of the same type). add the following functions to file ShapeDataObj.js makeRectangleByCenter, makeEllipse drawEllipse, makeTextLocationRectangle, makeTextLocationEllipse update drawRectangle by adding the case which is given the rotating angle and rectangle center.",10 DM-4486,"FITS Visualizer porting: Mouse Readout: part 1: projection","show the fits XY readout so that it update the position in the users selected coordinate system. The readout should also show the plot title. Write the dialog to change the coordinate system readout. Also show the pixel size and write the dialog that will change it between pixel size and screen pixel size. [1/28/16] Move this ticket to the next sprint. I cannot get it done in this sprint because # Spent time to work on other two tickets # Take off from work # It takes more time than estimates since I am not familiar with reducer and store etc. More study is needed. ",15 DM-4487,"FITS Save Dialog",NULL,10 DM-4488,"FITS Visualizer porting: Rotate",NULL,2 DM-4489,"FITS Visualizer porting: North/East Arrow","Add the north and east arrow like the gwt system has.",10 DM-4490,"FITS Visualizer porting: Layer Control Popup",NULL,6 DM-4491,"FITS Visualizer porting: Stretch Pulldown",NULL,1 DM-4492,"FITS Visualizer porting: Color bar pulldown",NULL,1 DM-4493,"FITS Visualizer porting: Restore to defaults & re-center",NULL,1 DM-4494,"FITS Visualizer porting: Show FITS Header","Task involves several steps: * Server side: VisServerCommands.Header needs to change to check to the JSON_DEEP parameter. In this case the return from VisServerOps.getFitsHeaderInfo should be converted into a format that the new javascript tables should understand (see loi how to get this format). Look at VisServerCommands.AreaStat for an example. * Client side: a call to the server: need to add getFitsHeaderInfo into PlotServiceJson.js. For reference, look at other calls in PlotServiceJson.js and the java version of the getFitsHeaderInfo PlotServiceJson.java * Client side: when header toolbar button pushed then make the call to the server. * Client side: when server call promise is resolved then show a dialog with the table data. Remember 3 color images should have a tab per color.",6 DM-4495,"FITS Visualizer porting: Flip",NULL,1 DM-4496,"FITS Visualizer porting: Expanded View",NULL,4 DM-4497,"FITS Visualizer porting: Expanded Single",NULL,4 DM-4498,"FITS Visualizer porting: Expanded View : WCS Match",NULL,6 DM-4499,"FITS Visualizer porting: Expanded View: Grid",NULL,6 DM-4500,"FITS Visualizer porting: Crop",NULL,4 DM-4501,"Fit Visualizer porting: Select Area",NULL,4 DM-4502,"FITS Visualizer porting: Statistics - part 1","dialog only",4 DM-4503,"FITS Visualizer porting: selecting points of catalog from image view, showing selected points","able to draw a rectangle on the image, and select the catalog entries overlaid on the image",5 DM-4504,"FITS Visualizer porting: Image Select Panel/Dialog","Converting the image select dialog/panel is a very big job and should be to be broken up into several tickets: Each ticket should reference this ticket as the base. Panel includes the following: * issa, 2mass, wise, dss, sdss tabs * file upload tab, upload widget might have to be written * url tab * blank image tab * target info reusable widget * 3 color support - any panel should show for 3 times, for read, green, and blue in 3 color mode * must be able to appear in a panel or dialog * must add or modify a plot * Allow to create version with most or less than the standard tabs. example - see existing wise 3 color or finder chart 3 color * A plot might need to be tied to specific type of image select dialog, we need a way to tie a plotId to and non-standard image select panel.",1 DM-4506,"Fit Visualizer porting: Thumbnail",NULL,3 DM-4507,"Fit Visualizer porting: Magnifier",NULL,2 DM-4508,"Experimentation and testing of new SuperTask infrastructure","WBS deliberately left unset as this is tracking LOE work.",6 DM-4509,"Migrate prototype SuperTask code to upstream repository","The prototype SuperTask code is now ready to be moved from the experimental repo to the upstream {{pipe_base}} repo. This requires the commits to be squashed and tidied.",6 DM-4510,"makeDocs uses old style python","{{makeDocs}} is written in python 2.4 style. This ticket is for updating it to python 2.7.",1 DM-4511,"Improve reStructuredText documentation","Enhance docs by covering - Images as links - Table spans - Abbreviations - :file: semantics, etc.",2 DM-4513,"Quoting of paths in doxygen configuration files breaks makeDocs","In DM-3200 I modified {{sconsUtils}} such that all the paths used in {{doxygen.conf}} files were quoted so that spaces could be used. This change broke documentation building (DM-4310) because {{makeDocs}} did not expect double quotes to be relevant. This ticket is to fix {{makeDocs}} and to re-enable quoting of paths in config files.",4 DM-4514,"Assess DECam ISR up to currently implemented","Not all known ISR corrections are applied or implemented to DECam data yet. For example, no cross-talk, edge-bleed, non-linearity, sky pattern removal, satellite trail masking, brighter-fatter, or illumination correction. But we have most of the basic ISR already. With what we already have, identify issues that would severely affect the quality of post-ISR processing.",3 DM-4515,"Flag out the glowing edges of DECam CCDs","Pixels near the edges of the DECam CCDs are bigger/brighter and correcting them is not trivial. One way to move forward is to mask them out. DESDM and CP mask 15 pixels on each edge. The cut was later raised to 25 pixels, with the inner 10 pixels flagged as SUSPECT. ",5 DM-4523,"Fix startup.py inside Docker container","qserv tag should be replace with qserv_latest",1 DM-4524,"Margaret's mgmt. activities in November","DMLT @ Princeton Weekly DMLT and Standups Local meetings TPR etc.",18 DM-4525,"Ops Planning and TOWG attendance - November","TOWG meetings, review service catalog as input to LOPT, review draft of operations WBS with Don and Athol",2 DM-4526,"L1 design specification and planning - November","With Steve, Jim, Don, and Jason, detailed design construction and planning for development of the Image Ingest and Processing System. Worked through LDM-230 and OCS design docs, input from discussions on Confluence. Made cleaner drawings (draft) and expanded Project plan. The plan is currently a ""to do"" list, without schedule or resource loading yet. ",10 DM-4527,"Facility Coordination Meeting and JCC meeting @ NCSA","Day 1 : Facility Coordination Day with Argonne, CC-IN2P3, NERSC, and NCSA Day 2: Extended JCC meeting Notes posted on Confluence: https://confluence.lsstcorp.org/pages/viewpage.action?spaceKey=JCC&title=Extended+JCC+meeting+--+2015-11-23 ",4 DM-4529,"Compilation errors from CLang (Apple LLVM 7.0) in XCode 7 on MacOSX","Compiling on MacOSX Yosemite with XCode 7, a number of files fail compilation. ---- {{core/modules/util/EventThread.h,cc}} fails because {{uint}} is used as a data type. This is non-standard (though some compilers support it), and should be replaced with {{unsigned int}}. ---- {{core/modules/wbase/SendChannel.h,cc}} fails because {{#include <functional>}} is missing. ---- {{core/modules/wsched/ChunkState.cc}} fails because {{#include <iostream>}} is missing. ---- {{build/qmeta/qmetaLib_wrap.cc}} (generated by SWIG) fails with many errors because the {{typedef unsigned long int uint64_t}} included in {{qmetaLib.i}} conflicts with MacOSX's typedef of it as {{unsigned long long}}.",1 DM-4534,"shellcheck linting of lsstsw bash scripts","This issue is to recover a branch from DM-4113 that was not merged due to issues with installing shellcheck under travis.",1 DM-4536,"Create release notes docs template","Create a template for release notes/other release-type documentation in the new docs.",2 DM-4538,"Research simulation tools","We need to do simulations of DCR and other effects when determining appropriate mitigation techniques. This will require settling on a simulation tool for doing this. The obvious choices are: phosim, galsim, and a roll your own solution. Look into which of these is the most reasonable choice and make a recommendation.",4 DM-4540,"Roll Qserv into SQuaRE release - part I","Improvements to codekit to support release process. ",4 DM-4544,"Revisit short and long term plans for butler","Revisit short and long term requirements and needs and capture it through stories.",5 DM-4545,"Review of storage quotes","Reviewed 4 products from 3 vendors with storage group. Prices and features comparison along with discussions on whether to integrate into condo or not. Also, discussed security considerations. Preliminary vendor/product chosen. Follow up questions sent to vendor. Will review with storage before we are able to purchase.",2 DM-4546,"Update to Sizing Model","Removed 'memory effectiveness' factor as it was already included in compute efficiency Updated capacity / tape to follow LTO roadmap Updated bandwidth / tape drive to follow LTO roadmap Changed ""tape!Number of new tapes purchased"" to round up to an even number for the archive site Changed ""tape!Number of new tapes needed"" to round up for both sites Changed ""tape!Number of tape drives needed based on bandwidth"" to round up for both sites Fixed 'tape!tape bandwidth' colums to take into account mixed tape drive types Updated 'tape!HPSS' to take into account retiring mover nodes Also began work with Spectra Logic to further improve tape predictions.",2 DM-4547,"Power requirements and LSST footprint at NCSA","Finalized power requirements with the UofI engineer. I plan to distribute verification compute across multiple racks in order to reduce per-rack power requirements and reduce per-rack network port counts. This allows us to drop from 3x 60A for the verification cluster to 3x 30A which is the same for the other racks. This will result in some cost savings and simpler planning for future use of the racks. Also, provided a plan for LSST's footprint in NPCF until 2032. We now have space reserved from the south side of Roger to the north side of Blue Waters. This space should be very visible from the room camera I believe. ",2 DM-4553,"Move butler from daf_persistence to daf_butler",NULL,6 DM-4555,"Add initial butler support for remote GET","For Get: If the mapper returns a URL: retrieve the URL contents into a file return the path to the file. This will be optimized in subsequent stories by ""add read support for various transports to the afw object readers"". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol. Need to solve the cleanup problem of when to delete the file that was downloaded. For Put: serialize the object to a temporary file transfer the file to the URL",10 DM-4556,"Fix docker workflow","Some issues where discovered while trying to package DM-2699 in Docker (for IN2P3 cluster deployment), they're fixed here. - apt-get update times out: why? - git clone then pull is too weak (if building the first clone fails, pull never occurs) => step merged - eupspkg -er build creates lib/python in /qserv/stack/.../qserv/... and next install can't remove it for unknow reason => build and install merged.",2 DM-4557,"Base site additions to simulator","Started using login on the Nebula cluster - set up some instances and used snapshot facility. Detected an error with launching from volume. Met with Chris Lindsey about issue. Started coding replicator and Foreman functionality. Began message taxonomy for comm between Foreman and Replicator. Installed log rotate, Rabbitmq, pip, pika and GCC on running instance. Successfully tested rabbitmq with another instance which had pika libs installed. Finished up prototyping of replicator and foreman code. Installed Nova Client on foreman instance so it can start and stop replicator instances as needed at runtime. Tested. Began a project to simulate realistic camera data from LSST to be used for various timings of DMCS prototype code. This was also a concentrated effort to learn about the nature of astronomical data, how it is represented, its various types of data, and practice with the C and Python libraries for building FITS files. Two sample files were built: One additively built and converted 16bit DES data into scaled 18bit data, then placed that binary representation in a 32bit integer, and the other was additively built up from PhoSim data. Files were built to represent a single LSST camera raft with 9 HDUs - one for each CCD. Files were over 600Mg. The code used in these files has been saved and parts turned into scripts for generating other sample files. Set up an environment for timing the imprint of these files as they are built and processed through the Base DMCS pipeline code. Initial timing was done on compression techniques. Wrote result paper on FITS file generation and timing. Errors due primarily to my own ignorance of this type of file format and how the data was truly represented were encountered several times. This project was personally extremely helpful in beginning to learn the vast amount of domain knowledge needed to complete future pieces of the L1 Base site code package. Added coding for setting up raw data to be moved and assembled into fits files in DMCS , simulating part of the Base DMCS data flow, into existing Base replicator/foreman prototype. ",70 DM-4558,"Functional drawings, specification writing, and info gathering ","Initial meeting with Don about spec work Began gathering architecture ideas for Base DMCS 4 Discussed requirements gathering for Base site network operations with Steve P., Paul, and Don. 1 Began specification draft for base site, and integrated drafts into project wiki 4 More spec ideas and posted them on Dons pages under the OCS Bridge page. 4 DMCS Planning meetings begin in earnest. 16 Planned DMCS diagrams for L1 base site and NCSA site as a group and drew them in support of Margaret’s trip to DM meeting. These were refactored a couple of times and are in confluence. 8 Camera meeting: about 1",38 DM-4559,"Bug fix and improvement for DECam processing","- Bug fix in DecamMapper policy of fringe dataset - Improve readme documentation about ingesting and processing raw data - Bug fix on translating Community Pipeline's Bad Pixel Mask (BPM) --- Previously in DM-4191 I looked up the wrong table for the BPM bit definition. - Flag the potentially problematic edge pixels as SUSPECT (DM-4515) - Add data products for coaddition processing",6 DM-4560,"Local LSST Sec Meeting",NULL,1 DM-4561,"Local LSST IaM Meeting",NULL,1 DM-4562,"Security Playbook",NULL,2 DM-4563,"Security Plan Renewal",NULL,3 DM-4564,"Convert basic table functionalities to JS.","Task includes server-side json conversion, data modeling, and a simple React table for presentation.",20 DM-4565,"Table (JS): selection feature.","This task is composed of: - converting java class SelectionInfo. - reducing data into its table model state - rendering SelectionInfo onto the TablePanel - creating action and action creator",6 DM-4566,"Upgrade to the latest react-highcharts library","We need to upgrade from the early version of react-highcharts to the latest one, compatible with React 0.14.3. Just switching to the new library does not work, need to resolve issues.",6 DM-4567,"Table (JS): large table handling","This task is composed of: - creating and adding a paging toolbar to TablePanel - adding external data loading feature to TablePanel - creating prefetch and background data fetching mechanism - use websocket events for reporting background statuses - requires new server-side code. - depends on DM-4578 - Integrate websocket messaging into flux",14 DM-4568,"Table (JS): sorting","This task is composed of: - introduce sorting feature into TablePanel - creating action and action creator - reducing data into its table model state",4 DM-4569,"Histogram View of a Table","Combine HistogramOptions and Histogram widgets into a Histogram viewer: - define histogram state tree, actions (getting/updating table statistics, getting/updating column data), and reducers - write a smart widget, which shows options and histogram side-by-side",10 DM-4570,"Table (JS): filtering","This task is composed of: - adding filter toolbar into TablePanel - filter validation syntax - creating action, action creator, and reducing data into its table model state - -generating meta info for enumerated columns- not sure if we wanted this. Also, added actOn feature to FieldInput.",6 DM-4571,"Suggestion Box widget","We need to find or implement a suggestion box widget in JS. Currently, it is used to suggest table column names in XY plot and in some forms.",6 DM-4572,"Table (JS): table options","This task is composed of: - adding table options panel to TablePanel. - providing features: - show/hide units in header - show/hide columns, reset to defaults, etc - page size",5 DM-4573,"JS expression parsing library","Since we are allowing column expressions we need a way to validate them on client side.",6 DM-4574,"Table (JS): text view","This task is composed of: - adding text view option to TablePanel",2 DM-4575,"XY Plot view of the table (JS): define state tree","Define state tree, actions, and reducers for XY Plot view of the table.",4 DM-4576,"XY Scatter Plot (JS) ","Implement basic scatter plot widget using react-highcharts library",8 DM-4577,"XY Scatter Plot Options (JS)",NULL,10 DM-4578,"Integrate websocket messaging into flux","This task is composed of: - design and implement messaging concept into flux Implementation thoughts: - convert inbound messages into actions - convert selected actions into outbound messages - add message action reducer with the concept of a message consumer. - consumer can be a predefined action creator or a function - allow consumers to be added/removed into/from the system after bootstrap",10 DM-4579,"XY Plot view of a table (JS)","Implement smart widget which shows toolbar, plot options, and XY plot.",10 DM-4580,"XY Plot view of a table (JS) - Toolbar","Toolbar, which toggles plot options, selection and filter buttons Extra: - handling zoom from the toolbar rather than using built-in zoom - ability to switch between histogram and scatter plot view",8 DM-4581,"XY Plot Viewer (JS) - density plot","Implement density plot using react-highchart library (Highchart's canvas-based heat map).",12 DM-4582,"XY Plot View of a table (JS) - selection support","Show/change selected/highlighted points. Ideally, this should be done without redrawing the whole plot. ",8 DM-4583,"SUIT: search returning images in a directory","- Create a sample search processor, which returns images in a given directory. - It should be using an external python task - Update search form configuration to use this search processor to return image metadata",2 DM-4584,"XY plot view of a table (JS) - density plot zoom support","Density plot zooming requires server call.",6 DM-4585,"XY Plot view of a table (JS) - density plot selection support (?)","density plot - how do we support selection? (In current version we turn off selection support when the plot is density plot)",6 DM-4586,"GWT Conversion: Login","This task is composed of: - adding user info into banner - includes user name and links for login, logout, and profile. - convert server-side code to return json - use messaging to handle current user state. - depends on DM-4578 Integrate websocket messaging into flux",4 DM-4587,"GWT Conversion: Search Panel","This task is composed of: - converting server-side code to return json - defining and loading search info data into application state - loading should be implemented so that it can be from a server fetch or a client declaration - creating action, action creator and reducing functions - rendering SearchPanel from search info data - attach SearchPanel to application: depends on GWT Conversion: layout",12 DM-4588,"Create ctrl_platform_nebula package to exercise ctrl_orca orchestration on Nebula","We create a ctrl_platform_nebula package to support processing with the LSST framework as orchestrated by the ctrl_orca/ctrl_execute packages within a HTCondor pool that resides on the Nebula OpenStack.",18 DM-4589,"GWT Conversion: basic layout for results","This task is composed of: - creating a results container that handle the layout of its components - define and load layout info into application state - creating action, action creator, and reducing functions - components include: - vis toolbar - last searched description - layout options: tri-view. side-by-side, single and popout can be added at a later time. - tables, image plots, xy plots. - depends on DM-4590: GWT Conversion: advance resizable layout panel",4 DM-4590,"GWT Conversion: advance resizable layout panel","Create an advance React component for layout. Features should include: - a set of predefined layouts - resize strategies - generic for reuse",6 DM-4591,"GWT conversion: System notifications","This task is composed of: - adding notification panel to the application - convert server-side code to use messaging for notifications - use messaging on client-side to handle notifications - creating action, action creator, and reducing functions - depends on DM-4578 Integrate websocket messaging into flux ",3 DM-4595,"GWT Conversion: History and routing","First pass at the implementation of history and routing. Define a framework in which the application can be: - bookmarked - record state in history - retore application from a url ",4 DM-4596,"Remove deprecated versions of warpExposure and warpImage","afw.math supports two templated variants of warpExposure and warpImage, one that takes a warping control and the other which does not. The latter have been deprecated for a long time and are no longer used. I think it is time to remove them.",1 DM-4601,"Build docs.lsst.io Doc Index Page","Create an HTML landing page for all DM documentation/documents - Software Documentation - Developer Guides - Requirement and Design Documentation - Technotes - Papers - Presentations The page will be implemented as a static site. The page build will be template driven, with content scraped from the metadata.yaml resources of technotes (among other sources). Since this is the first SQuaRE web project, this ticket will also involve effort in establishing a CSS+HTML pattern library and Gulp-based development workflows. Long term, this investment will be returned with new dm.lsst.org, technote and Sphinx documentation web designs.",2 DM-4603,"sconsUtils tests should depend on shebang target","Some tests rely on code in the {{bin}} directory. Whilst these tests have been modified to use {{bin.src}} the general feeling is that the test code should be able to rely upon the {{shebang}} target having been executed before they are run.",1 DM-4604,"make codekit repos.yaml aware","Up to now codekit assumed the repo name is the same as the eups package name. FIx it by using repos.yaml. ",1 DM-4609,"Partition package should use the standard package layout","The partition package does not build on OS X El Capitan because the package is not laid out in the standard manner and whilst {{sconsUtils}} is used most of the default behaviors are over-ridden. This means that fixes implemented for DM-3200 do not migrate over to {{partition}}. I think the best approach would be to reorganize the package so that it does build in the normal way.",1 DM-4610,"Research Kerberos and LDAP replication options","IAM components, including the Kerberos KDC, need to be replicated between NCSA and Chile machine rooms. This may impact whether LSST can use NCSA's production Kerberos instance (if it supports selective replication) or needs a separate Kerberos instance that can be replicated outside NCSA. This task is to research and document the options, in consultation with NCSA Kerberos admins, and propose a Kerberos replication approach.",1 DM-4611,"Receive, verify and test network equipment",NULL,4 DM-4612,"Install networking hardware into openstack and verify operation",NULL,4 DM-4613,"Verify Network Emulator operation",NULL,4 DM-4614,"Design network emulator integration into workflow",NULL,2 DM-4615,"Deploy heirarchical queuing to test image precedence","Once we have a working workflow ready to move data between the ""base site"" and ""archive site"" (both at NCSA), deploy a base site exit router with stacked queuing to test prioritization of image traffic over various network conditions.",8 DM-4616,"Migrate Qserv code to stream-based logging","Migrate Qserv code from LOGF_* to LOGS_*. While doing it, we will also revisit logging levels: in particular we are abusing INFO, most of what is now in INFO should be on DEBUG, in some places where INFO is used to cover unusual conditions, it should go to WARNING. Further, we will unify how we initialize logging structures. Per discussions at 2015/12/09 Qserv meeting, we like best {code}LOG_LOGGER _log = LOG_GET(""lsst.qserv.<module>.<file>""){code} in anonymous namespace in cc files. Logging from .hh files is strongly discouraged. This involves changing ~600 places.",10 DM-4617,"Send all chunk-queries to primary copy of the chunk","We are planning to distribute chunks / replicas across worker nodes such that each node will have a mix of primary copies for some chunks, and backup copies for some chunks. While doing shared scan, we are going to always rely on the primary chunks (e.g., all queries that need a given chunk should be sent to the same machine so that we read that chunks only once on one node). This story involves tweaking xrootd to ensure we don't send chunk-queries to nodes hosting non-primary copies.",5 DM-4625,"Design for butler support of multiple repositories","Work on design for Gregory/SUI's request: We need to understand how put()/writing works when multiple repositories are made visible through a single Butler. For get()/reading a single search order makes sense. For put() it may be desirable to support alternative destinations (local disk, user workspace, Level 3 DB) or even multiple destinations for a single put().",6 DM-4628,"Explore coadd processing with DECam data with default config","Starting with raw DECam data, perform single frame processing and then try image coaddition with a few visits of images. ",6 DM-4630,"Provide input to CalibrateTask design work","Provide requirements and advice as input to the effort to redesign CalibrateTask (DM-3881).",10 DM-4631,"Create IDL pipeline workflow for DRP processing - processCcdDecam","For the verification datasets work we need the ability to run a dataset all the way through DRP processing that takes advantage of many cores (orchestration), keeps track of successful and failed dataIDs, creates individual log files, and creates helpful QA plots/metrics/webpages. Nidever will use his PHOTRED IDL workflow and rewrite it for the stack. The first step is processCcdDecam.",5 DM-4632,"Create IDL pipeline workflow for DRP processing - makeCoaddTempExp","For the verification datasets work we need the ability to run a dataset all the way through DRP processing that takes advantage of many cores (orchestration), keeps track of successful and failed dataIDs, creates individual log files, and creates helpful QA plots/metrics/webpages. Nidever will use his PHOTRED IDL workflow and rewrite it for the stack. processCcdDecam is working. The next step is makeCoaddTempExp. ",4 DM-4637,"lsstsw should symlink afwdata or allow an envvar","To reduce disk usage, it is very handy to be able to make build/afwdata and stack/afwdata/BLAH be symlinks. build/ is easy: just make the symlink and then never have touch it unless you rm your whole stack. stack/afwdata/BLAH is harder: each time you rebuild something that depends on afwdata, it will install a new copy of afwdata, which you'll have to manually remove and declare your symlink with a new tag. A couple of ways to make this more automatic: * lsstsw checks whether build/afwdata is a symlink, and if so just makes the new stack ""install"" a symlink to the same directory. * Check for some environment variable (e.g. AFWDATA_BASE_DIR or something) and if that exists, just make a symlink to it, or make a dummy eups table that points to that directory and don't put anything in stack at all.",5 DM-4639,"modernize afw code and reduce doxygen errors","I would like to make some simple modernizations afw code and reduce doxygen warnings as much as practical. The modernizations I had in mind were: - Move doc strings from .cc files to .h files and standardize their format - Use {{namespace lsst { namespace afw { ...}} in .cc files to make the code easier to read - Eliminate all {{<Class>::Ptr}} and {{<Class>::ConstPtr}} typedefs (replacing with {{PTR(<Class>)}} and {{CONST_PTR(<Class>)}}). - Make sure .py files import the appropriate packages from future and (where practical) pass the flake8 linter Regarding doxygen warnings: I think moving the documentation to .h files will help in many cases. Some warnings may be impractical to fix, such as complaining about not documenting ""cls"" for python class methods. ",6 DM-4641,"Implement mouse interaction with the drawing infrastructure",NULL,8 DM-4642,"Migrate scisql and mysqlproxy to mariadb","MySQLproxy and SciSQL relies on MySQL, they should now move to MariaDB",4 DM-4643,"Add utility function to handle client-side download requests.","Create utility function to handle client-side download requests. It needs to be done in a way that does not mess with history and current page state.",1 DM-4644,"Add workflow code to lsst-dm github","Since we have split the code for Super Task, all the workflow code should go in a different repository",2 DM-4645,L1-CONOPS,"Contribute to L1-CONOPS document",10 DM-4647,"investigate replicating EUPS published packages","(This ticket is for work that has already been done, per internal discussion in SQRE, but accidentally without an open ticket) https://github.com/lsst-sqre/lsyncd-eupspkg https://github.com/lsst-sqre/sandbox-pkg",4 DM-4648,"Support sqlalchemy use with qserv","When one tries to connect to qserv using sqlalchemy there is an exception generated currently: {noformat} $ python -c 'import sqlalchemy; sqlalchemy.create_engine(""mysql+mysqldb://qsmaster@127.0.0.1:4040/test"").connect()' /u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py:298: SAWarning: Exception attempting to detect unicode returns: InterfaceError(""(_mysql_exceptions.InterfaceError) (-1, 'error totally whack')"",) ""detect unicode returns: %r"" % de) Traceback (most recent call last): File ""<string>"", line 1, in <module> File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py"", line 2018, in connect return self._connection_cls(self, **kwargs) File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py"", line 72, in __init__ if connection is not None else engine.raw_connection() File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py"", line 2104, in raw_connection self.pool.unique_connection, _connection) File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py"", line 2078, in _wrap_pool_connect e, dialect, self) File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py"", line 1405, in _handle_dbapi_exception_noconnection exc_info File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/util/compat.py"", line 199, in raise_from_cause reraise(type(exception), exception, tb=exc_tb) File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py"", line 2074, in _wrap_pool_connect return fn() File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/pool.py"", line 318, in unique_connection return _ConnectionFairy._checkout(self) File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/pool.py"", line 713, in _checkout fairy = _ConnectionRecord.checkout(pool) File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/pool.py"", line 480, in checkout rec = pool._do_get() File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/pool.py"", line 1060, in _do_get self._dec_overflow() File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/util/langhelpers.py"", line 60, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/pool.py"", line 1057, in _do_get return self._create_connection() File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/pool.py"", line 323, in _create_connection return _ConnectionRecord(self) File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/pool.py"", line 454, in __init__ exec_once(self.connection, self) File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/event/attr.py"", line 246, in exec_once self(*args, **kw) File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/event/attr.py"", line 256, in __call__ fn(*args, **kw) File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/util/langhelpers.py"", line 1312, in go return once_fn(*arg, **kw) File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/engine/strategies.py"", line 165, in first_connect dialect.initialize(c) File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/dialects/mysql/base.py"", line 2626, in initialize default.DefaultDialect.initialize(self, connection) File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py"", line 256, in initialize self._check_unicode_description(connection): File ""/u2/salnikov/STACK/Linux64/sqlalchemy/2015_10.0/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py"", line 343, in _check_unicode_description ]).compile(dialect=self) File ""/u2/salnikov/STACK/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/cursors.py"", line 174, in execute self.errorhandler(self, exc, value) File ""/u2/salnikov/STACK/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/connections.py"", line 36, in defaulterrorhandler raise errorclass, errorvalue sqlalchemy.exc.InterfaceError: (_mysql_exceptions.InterfaceError) (-1, 'error totally whack') {noformat} The reason for that is that sqlalchemy generate few SELECT queries to figure out unicode support by the engine, and those selects are passed to qserv which cannot parse them. Here is the list of SELECTs which appears in proxy log: {code:sql} SELECT CAST('test plain returns' AS CHAR(60)) AS anon_1 SELECT CAST('test unicode returns' AS CHAR(60)) AS anon_1 SELECT CAST('test collated returns' AS CHAR CHARACTER SET utf8) COLLATE utf8_bin AS anon_1 SELECT 'x' AS some_label {code}",3 DM-4649,"Create and rename the sims_dustmaps repository to sims_maps","Create and rename the sims_dustmaps repository to sims_maps. This is my plan after talking to [~jhoblitt]: Add sims_maps to {{lsst_build/repos.yml}}. Change related dependencies and create ticket branches, run CI to confirm the changes.",2 DM-4650,"Update sims_dustmaps/sims_maps repository to use git-lfs","Update sims_dustmaps to use git-lfs.",1 DM-4651,"Convert GWT code to pure JavaScript (X16)","We plan to continue the GWT to JS conversion in Summer 16. The goal is to finish it.",100 DM-4652,"CI debugging","diagnosing build failures and refreshing build slaves",1 DM-4653,"Ci Deploy and Distribution Improvements part III",NULL,60 DM-4656,"Port code style guidelines to new DM Developer Guide","Verbatim port of DM Coding style guidelines to Sphinx doc platform from Confluence. - https://confluence.lsstcorp.org/display/LDMDG/DM+Coding+Style+Policy - https://confluence.lsstcorp.org/display/LDMDG/Python+Coding+Standard - https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908666 and contents I’m unclear whether these pages should be included: - https://confluence.lsstcorp.org/pages/viewpage.action?pageId=20283399 (C++ ‘using’) - https://confluence.lsstcorp.org/pages/viewpage.action?pageId=20284190 (how to use C++ templates) - https://confluence.lsstcorp.org/pages/viewpage.action?pageId=20283399 (C++11/14; which should seem to belong in the code style guide) Any temptation to amend and update the style guideline content will be avoided.",5 DM-4657,"Port RFC/RFD/Decision Making Page to new docs","Port to new Sphinx docs: https://confluence.lsstcorp.org/display/LDMDG/Discussion+and+Decision+Making+Process?src=contextnavpagetreemode",3 DM-4665,"Read and extend SuperTask technical note","Read the current version of DMTN-002 and comment. Write new sections describing the overall architecture and the expected role of SuperTask in the system.",3 DM-4666,"Review existing CmdLineTask instances' inputs and outputs","Review most or all existing DM CmdLineTask subclasses to understand their external inputs and outputs. This will inform the design of the successor to the interim SuperTask.execute( DataRef ) interface. The issue is that the single-DataRef interface supports only 1:1 input:output relationships, or N:1 relationships where a list of inputs is derivable from an output dataid. This is believed to be insufficiently general.",6 DM-4667,"Improve sphgeom documentation","Per RFC-117, the sphgeom package needs decent overview documentation, linked from the top-level README.md. The doxygen output should also be reviewed.",2 DM-4668,"audit obs_subaru defaults and move them to lower-level code","The obs_subaru config overrides contain many useful settings that aren't actually specific to HSC or Suprimecam. These should be moved down to the low-level defaults in the config classes themselves, so new obs_ packages don't have to copy these configurations explicitly.",2 DM-4671,"configure WebDAV with Kerberos/LDAP on lsst-auth1","Configure WebDAV for Kerberos authentication and LDAP authorization. Create example subdirectories where LDAP groups determine access (using .htaccess files): * lsst: anyone in lsst group can read/write to this directory * ncsa: anyone in all_ncsa_employe can write, anyone in lsst can read",6 DM-4672,"IAM process for managing L3 data access","Document a process for managing access to L3 Data Products. On the wiki: https://confluence.lsstcorp.org/display/LAAIM/Managing+L3+Data+Access",1 DM-4673,"Prototype LSST User/Group Manager",NULL,1 DM-4677,"Design Interfaces for Memory Management for Shared Scans","Part of the shared scans involve memory management - a system that will be used by Qserv that will manage memory allocation / pin chunks in memory. This story involves designing the API between Qserv and the memory management system. ",8 DM-4678,"qserv/cfg has to be removed by ""scons -c""","qserv-meta.cong was still pointing on MySQL instead of MariaDB, even after running ""scons -c"". This error-prone behaviour should be fixed.",2 DM-4679,"work with database team to exercise all the APIs for data access (F16)","SUI will continue to work with database team to exercise all the APIs for data access. All known issues should be worked out in S16 cycle.",40 DM-4680,"Provide API for tabular data display using Firefly","We need to provide JavaScript API access to all the table displaying features to give user more control when using Firefly API to displaying table data in their own web page or to build customized web UI ",10 DM-4681,"Provide a prototype version of LSST web UI ","SUIT deployment at NCSA to access SDSS strip82 data processed by DM stack in 2013. * Use the data access API, or TAP API * Light curve for time series data * Connection between the light curve data point and the image that the data point coming from ",60 DM-4682,"Implementation of multiple repositories v1",NULL,25 DM-4683,"Implementation of multiple repositories v2",NULL,15 DM-4688,"Changed the implementation of HistogramProcessorTest due to the minor change about the algorithm in the HistogramProcessor","In Histogram, when the data points fall on the bin edges, the following rules are used: # For each bin, it contains the data points fall inside the bin and the data point fall on the left edge. For example, if binSize=2, the bin[0] is in the range of [0,2]. The data value 0 is in bin[0] . # For each bin, the data point falls on the right edge is not included in the number point count. For example if binSize=2, the bin[0] is having the range of [0,2]. The data value 0 is in bin[0] but the data value 2 is not in the bin[0]. # For the last bin, the data points fall inside the bin or fall on the left or right bin are counted as the number of bin points. The last rule is newly introduced. ",2 DM-4689,"Firefly visualization Java/JS code refactoring and bug fixes(F16) ","This epic will capture all the Java and JS code refactoring in Firefly, bug fixes, JS code optimization and performance enhancement. ",80 DM-4690,"Design worker scheduler for shared scans",NULL,8 DM-4691,"Data Distrib proto (Jan)",NULL,25 DM-4692,"Refactor ProcessCcdTask and sub-tasks","Based on conversation spurred by DM-3881 as discussed [on clo|https://community.lsst.org/t/requirements-for-overhauled-calibration-task/370], this ticket will refactor ProcessCcdTask to be easier to extend and instrument, easier to understand, and more modular. The main work will be to break up ProcessCcdTask into it's component modules and, and reconfigured to meet the requirements as outlined by the clo discussion.",16 DM-4694,"distributed loader",NULL,12 DM-4696,"Implement memory mgmt for shared scans",NULL,12 DM-4697,"Implement worker scheduler for shared scans",NULL,20 DM-4698,"Add initial butler support for remote PUT","For Get: If the mapper returns a URL: retrieve the URL contents into a file return the path to the file. This will be optimized in subsequent stories by ""add read support for various transports to the afw object readers"". This is a degenerate case that will be used if the object does not have a reader for a given transport protocol. Need to solve the cleanup problem of when to delete the file that was downloaded. For Put: serialize the object to a temporary file transfer the file to the URL",10 DM-4699,"Create ALERT framework for qserv","At the moment Qserv code will throw exception when something wrong/unusual happens. That is not always the best idea to do in a server code that is needs to run 24x7. If we don't throw exception and just log the issue, it might get unnoticed in the log files. So, it'd be useful to have some sort of alert framework where we could send alerts when something strange / unexpected happens in Qserv code and we are able to ""ignore it"" and continue running the server. It can be as naive as writing to a special place, or sending an email, or messaging the DBA etc. This story involves designing and implementing such framework. The sooner we do it the better so that we don't accumulate new code that is throwing exceptions where it should not.",10 DM-4700,"Revisit Qserv code that throws exceptions","We have ~500 places where we throw exceptions in qserv/core/modules/*/*. Revisit all of them and make sure we catch these exceptions properly.",15 DM-4701,"Promote IsrTask to command line task.","As pointed out by [~nidever] in DM-4635, it would be quite useful to have the IsrTask callable as a command line task without having to do all the other steps in ProcessCoaddTask.",4 DM-4702,"Promote CharacterizationTask to command line task","In refactoring the processCcd.py script, we'd like to make each component callable by command line as well. This is to promote the image characterization task to a command line task. A requirement will be that this task be able to run on data without IsrTask having been run (command line tasks should be insulated as much as possible from knowing about previous processing).",4 DM-4703,"Promote CalibrateTask to command line task","The task that takes care of measurement and calibration on characterized images will be promoted to a command line task. As with the other command line tasks, it should be possible to run the calibration and measurement command line task on data without necessarily running IsrTask or CharacterizeTask. Of course, this means the task will have to get a PSF from somewhere, see [clo|https://community.lsst.org/t/requirements-for-overhauled-calibration-task/370] for suggestions.",4 DM-4704,"Qserv integration tests fail on CentOS7 with gcc 4.8.5","The version of gcc that ships with CentOS7, {{gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4)}}, appears to miscompile the qserv worker source in a way that makes it impossible to actually run queries. Installing {{devtoolset-3-toolchain}} and {{devtoolset-3-perftools}} via {{yum}} to get gcc 4.9 resolves the issue.",2 DM-4705,"qdisp/testQDisp fails with mariadb","Fabrice fried to build qserv with mariadb and it caused failure in one of the unit test: qdisp/testQDisp with the message: {noformat} pure virtual method called terminate called without an active exception {noformat} Runnig it with GDB it' obvious that there is a problem with resource lifetime management in qdisp/testQDisp.cc. The problem is that XrdSsiSessionMock is destroyed sooner than other objects that use it. One way to resolve this problem is to instantiate XrdSsiSessionMock earlier than other objects that use it (to reverse the order of destructors), possibly make it a global instance. Big mystery here is how mariadb could trigger this interesting behavior and why did not we see this earlier.",1 DM-4706,"Rerun and create a repository for CFHT astrometry test.","Understand, re-rerun, and recreate clean version of [~boutigny] 's CFHT astrometry test for the astrometry RMS for two sample CFHT observations. This test is on the NCSA machines in /lsst8/boutigny/valid_cfht Create a repository for this test with an eye toward it becoming integrated in a validation suite for the stack. ",1 DM-4707,"Adapt CFHT astrometry test for DECam COSMOS field validation","Adapt the CFHT astrometry validation test to the DECam reprocessing effort. Will focus on the repeat observations of the COSMOS field. Goal is to just do a simple two-observation comparison. Doing a full test of all of the observations will be a later story. ",1 DM-4708,"Integrate astrometry test into SDSS demo lsst_dm_stack_demo","Incorporate the astrometry test as an optional component in lsst_dm_stack_demo. This is chosen because lsst_dm_stack_demo currently serves as the very loose stack validation and understanding how to do astrometric repeatibility testing in this demo will help explore how it would make sense to put in a fuller CFHT validation test of the DM stack.",1 DM-4709,"Prototype a validation module of the stack using CFHT data.","Create a prototype standalone validation test of the astrometric performance of the stack on suitable CFHT data. Module is called `validate_drp` http://github.com/lsst/validate_drp_cfht Decide how those data should be provided (testdata_cfht being one obvious possibility), and determine if obs_cfht tests and the tests for this validate_drp module should use the same test datasets. This is prototyping for DM-2518.",1 DM-4710,"host identification info needs to be part of log message","The EventAppender needs to add host identification (host/process/id) information to the log message it transmits. This was inadvertently left out.",3 DM-4711,"Edit testdata_cfht to pass obs_cfht unit tests","This ticket covers the first half of the issues in DM-2917. {{testdata_cfht}} was left unedited while some past changes in {{obs_cfht}} {{MegacamMapper}} required coordinated changes. The goal of this ticket is to simply pass the unit tests currently in {{obs_cfht}}. ",1 DM-4712,"Fix documentation and restructure workflowTask",NULL,2 DM-4713,"Improve documentation on pipe_base/supertask",NULL,2 DM-4714,"Bad OpenBlas setting in miniconda/numpy causes very poor performance for running multiple processes","I have been running many processes of processCcdDecam.py on my new linux machine in Tucson (bambam). To my surprise, running 40 processes at once gets very poor performance (~70 sec per process) compared to running a single process (~16 sec). I expected some performance hit because of larger overheads but not a factor of 4! I ran it both on a spinning HDD and PCIe SSD but they both had the same problem. I also tried running it on multiple visits versus multiple chips for a single visit (all accessing the same MEF FITS file) but this made no difference. I tested it with various numbers of processes and found that the time per processes increases linearly with the number of processes running. [~jmatt] has been helping me track this down. We used some performance tools (htop, iotop, and perf) to figure out what was going on. It was clear that the issue was not a RAM or I/O problem. By watching htop while the 40 processes were running it became clear that once some of the processes hit ""deblending"" everything slowed down considerably and all cores were maxed out and showing lots of kernel traffic. I also ran processCcdDecam.py with deblending turned off and the performance was much more reasonable (~24 sec. per process). After more digging (with perftop), we found that there was a lot of swapping going on during the deblending step by ""openblas"". This is a package that numpy uses for speeding up certain computationally intensive tasks using multithreading (e.g. linear algebra). By default each openblas instance takes advantage of ALL cores on a machine. So all 40 processes were trying to use all available cores and most of the time was spent swapping between all of these threads. OpenBlas can be configured to use a more reasonable number of cores/threads, but the version that LSSTSW uses is installed by miniconda via a dependency of numpy and, as far as we could tell, it's not possible to configured NUM_THREADS for OpenBlas with miniconda. We ended up compiling our own version of OpenBlas with NUM_THREADS = 6 (the maximum threads that OpenBlas uses) and the performance was great, 24 sec. I'm not sure what the solution is for this but we probably don't want to go with miniconda for the default LSSTSW installation (uses currently done by bin/deploy). JMatt might have comments to add. ",4 DM-4716,"Track down reason for slow performance when running many jobs of processCcdDEcam on bambam","During the processing of the COSMOS data for the verification dataset work I ran many jobs of processCcdDecam.py on the new linux server, bambam. The performance was very slow, 4x longer than running a single job at a time. Figure out what is going on.",2 DM-4717,"Meetings Dec 2015","Verification dataset meetings, RFD meetings, DES Chicagoland meeting and preparation",6 DM-4718,"Other LOE -- Dec 2015","weekly LSST local grouop meetings, NCSA meetings (All-hands, software, etc), code review, other local meetings, postdoc meetings and tasks",6 DM-4719,"Vendor input on sizing predictions","Discussions about tape-pricing and disk-pricing predictions from Spectra and DDN respectively in order to improve our forecasting. This information needs to be incorporated into LDM-144.",2 DM-4720,"More preparation for FY16 hardware","More pricing iterations with several companies and incorporating that information into our final decision. More Q&A with storage companies re: comparable features. Compiled all storage option quotes into a spread sheet which now forms as a good comparison and helps LDM-144 forecasting. Awaiting quotes for racks and PDUs. Now that rack size is known for the Chilean DC, this will serve us well for LDM-144 costs. Power issues for FY16 hardware are settled and we are ready to schedule installation as soon as the hardware purchase contract is complete. Tagging issues for FY15 hardware complete. Looking into pre FY15 tagging. Requested that LSST/Aura perform an inventory request to complete the circle and prove the process. Working with Spectra / NetSource to create a sustainable tape condo that can serve LSST through 2030.",4 DM-4721,"Plan DM’s communication / documentation / information architecture strategy","Plan and write a technote outlining communication and documentation platforms from a DM perspective. The technote will specify - how each platform is used - what developments need to be done - address integrations with LSST-wide communications projects - address information architecture (generally, the ease of discovering the right information)",2 DM-4722,"File tickets for list of stack deficiencies and suggested upgrades","K-T suggested that I take my list of ""stack deficiencies and suggested improvements"" [https://confluence.lsstcorp.org/display/SQRE/Stack+Deficiencies+and+Suggested+Upgrades] on confluence and (with Tim J.'s help) create tickets for each item (as much as possible) so that the work could be scheduled. ",3 DM-4723,"Continue learning about middleware","Learn more about orchestration, task execution, and logging. ",3 DM-4724,"Implement zenodio.harvest","Harvest metadata about records in a Zenodo Community collection using the {{oai_datacite3}} format. See https://zenodo.org/dev Part of the [zenodio|https://github.com/lsst-sqre/zenodio] Python package. This tool will be used by our technote and the documentation indexing platforms.",3 DM-4728,"Doxygen package fails to build with flex 2.6","To wit: {code} $ flex --version flex 2.6.0 $ bash newinstall.sh LSST Software Stack Builder [...stuff...] eups distrib: Failed to build doxygen-1.8.5.eupspkg: Command: source /Users/jds/Projects/Astronomy/LSST/stack/eups/bin/setups.sh; export EUPS_PATH=/Users/jds/Projects/Astronomy/LSST/stack; (/Users/jds/Projects/Astronomy/LSST/stack/EupsBuildDir/DarwinX86/doxygen-1.8.5/build.sh) >> /Users/jds/Projects/Astronomy/LSST/stack/EupsBuildDir/DarwinX86/doxygen-1.8.5/build.log 2>&1 4>/Users/jds/Projects/Astronomy/LSST/stack/EupsBuildDir/DarwinX86/doxygen-1.8.5/build.msg exited with code 252 $ grep error /Users/jds/Projects/Astronomy/LSST/stack/EupsBuildDir/DarwinX86/doxygen-1.8.5/build.log commentscan.l:1064:55: error: use of undeclared identifier 'yy_current_buffer' commentscan.l:1126:58: error: use of undeclared identifier 'yy_current_buffer' {code} Builds fine using {{flex 2.5.35 Apple(flex-31)}}.",1 DM-4729,"HSC backport: Add functions to generate 'unpacked matches' in a Catalog","The qa analysis script under development (see DM-4393) calls to HSC {{hscPipeBase}}'s [matches.py|https://github.com/HyperSuprime-Cam/hscPipeBase/blob/master/python/hsc/pipe/base/matches.py] which adds functions to generate ""unpacked matches"" in a Catalog (and vice versa). It will be ported into {{lsst.afw.table}}. The port includes following HSC commits: *Add functions to generate 'unpacked matches' in a Catalog.* https://github.com/HyperSuprime-Cam/hscPipeBase/commit/210fcdc6e1d19219e2d9365adeefd9289b2e1186 *Adding check to prevent more obscure error.* https://github.com/HyperSuprime-Cam/hscPipeBase/commit/344a96de741cd5aafb5e368f7fa59fa248305af5 *Some little error handling helps.* https://github.com/HyperSuprime-Cam/hscPipeBase/commit/61cc053b873d42802581adff8cbbdb52a348879e (from branch: {{stage-ncsa-3}}) *matches: add ArrayI to list of field types that require a size* https://github.com/HyperSuprime-Cam/hscPipeBase/commit/d4ccd11d8afbcdd9cf0b35eba948cca4b5d09ba5 (from branch {{tickets/HSC-1228}}) Please also include a unittest.",3 DM-4730,"Adapt qa analysis script for LSST vs. HSC single visit processing comparison","The qa analysis script ported from HSC and adapted to LSST on DM-4393 currently performs qa on single visit processing by comparing outputs of a single run from the different measurement algorithms and comparing those with the astrometry/photemetry reference catalog. Here we will add functionality to directly compare two different runs of the same dataset (requiring accommodations for two butlers). Since the goal is to compare outputs from runs on the LSST vs. HSC stacks, this will require a mapping of the different schemas of the persisted source catalogs of the two stacks.",10 DM-4731,"Add labels to qa analysis plots for better interpretation","The plots output by the qa analysis script (see DM-4393) currently do not display any information regarding the selection/rejection criteria used in making the figures and computing the basic statistics. This includes magnitude and clipping thresholds. This information should be added to each plot such that the figures can be interpreted properly.",2 DM-4733,"lsst-build should support enabling Git LFS in an already-cloned repository","A repository which does not use Git LFS is created and described in {{repos.yaml}}. It runs through CI, and is cloned onto a Jenkins build slave. Subsequently, the repository configuration in {{repos.yaml}} is updated to enable LFS. The build system should notice this change and update the cloned repository on disk appropriately. Currently, it doesn't.",1 DM-4734,"afw fails to build on a machine with many cores","The afw package does not build reliably (if at all) on a linux box at UW (""magneto"", which has 32 cores and 128 Gb of RAM). The failure is that some unit tests fail with the following error: {code} OpenBLAS: pthread_creat error in blas_thread_init function. Error code:11 {code} For the record, /usr/include/bits/local_lim.h contains this: {code} /* The number of threads per process. */ #define _POSIX_THREAD_THREADS_MAX 64 /* We have no predefined limit on the number of threads. */ #undef PTHREAD_THREADS_MAX {code} It appears that the build system is trying to use too many threads when building afw, which presumably means it is trying to use too many cores. According to [~mjuric] the package responsible for this is {{eupspkg}}, and it tries to use all available cores. A workaround suggested by [~mjuric] is to set environment variable {{EUPSPKG_NJOBS}} to the max number of cores wanted. However, I suggest we fix our build system so that setting this variable is unnecessary. I suggest we hard-code an upper limit for now, though fancier logic is certainly possible. A related request is to document the environment variables that control our build system. I searched for {{NJOBS}} on confluence and found nothing.",1 DM-4735,"Remove dead code from configuration procedure"," - remove scratch db? - cleanup tmp/sql/*.sql filesi - remove xrootd configuration script if useless? - cleanup configuration script style (i.e. tmp/*.sh) ",3 DM-4736,"Study if mysqlproxy can be compatible with mariaDB client","mysqlproxy is not compliant with mariaDB client: see https://mariadb.com/kb/en/mariadb/mariadb-vs-mysql-compatibility/#incompatibilities-between-mariadb-and-mysql-proxy Nevertheless the trivial fix proposed (remove progress-report options) doesn't seems to work... {code:bash} mysql --host=127.0.0.1 --port=4040 --user=qsmaster --batch qservTest_case01_qserv ERROR 1043 (08S01): Bad handshake {code}",4 DM-4737,"Improve 'unit' tests using database","- Add mock database for it to work during unit tests or run it apart from unit tests? - Fix testLocalInfile (read configuration file) - Try all core/modules/sql/testSql* ",5 DM-4738," Improve LOAD LOCAL INFILE management on czar side","Option ""mysql_options( m, MYSQL_OPT_LOCAL_INFILE, 0 );"" is added to all C++ sql client instance due to common sql interface, but is only required on master (for merging results) is it possible: - to remove LOCAL keyword (on czar virtfile is on the same machine that mariadb server) - or to set it in Qserv czar/master configuration - or to set it in master MariaDB instance only?",4 DM-4739,"Update kernel on IN2P3 cluster","The ""Kernel Panic"" issue is non-blocking right-now for John due to machine automated reboot, but we have to solve it to target a stable production system. With Yvan, we're converging on next update for the cluster: - on my side I update Qserv metadata on ccqserv100 w.r.t. new Qserv metadata format, and then I test Docker+Qserv on ccqserv100->ccqserv124, - then CC-IN2P3 team launches a upgrade of the kernel to kernel-ml ( ""mainline stable"" branch of The Linux Kernel Archives) on ccqserv100->ccqserv124, this could be done in January, - I control Qserv behaviour is still the same than before, - then Qserv developpers can use this cluster to see if ""Kernel Panic' issue is solved. If it work we'll update to kernel-ml on ccqserv125->ccqserv149, if it doesn't, cc-in2p3 and Qserv developpers will have to find an other solution. Regards, Fabrice",5 DM-4740,"Audit and document obs_subaru scripts","{{obs_subaru}} has a {{bin.src}} directory containing a variety of miscellaneous scripts. Some of these may be actively useful; others could be useful, but require modernizing to work with the latest version of the LSST codebase; others are obsolete or duplicate functionality available elsewhere. Throughout, documentation is lacking. Please audit this directory: remove the scripts which are useless and ensure the others are working and properly documented.",5 DM-4741,"Cyber security infrastructure document","This document details anticipated security infrastructure and roles needed for the operations of LSST at the base and summit observatory site. ",1 DM-4742,"Bi-weekly LSST IaM meetings for December","Bi-weekly IaM meeting between NCSA and LSST for the month of December 2015. Local coordinating meetings also included.",1 DM-4743,"Make deblender more robust against weird PSF dimensions","[~boutigny] reports two problems with PSF dimension calculations in the deblender that result in fatal errors, because earlier checks for bad dimensions intended to cause more graceful failures are incomplete. The first appears to happen when the PSF dimensions are highly non-square, and the image width is smaller than 1.5x FWHM while the image height is more than 1.5x FWHM (or the opposite). {code:hide-linenum} measureCoaddSources FATAL: Failed on dataId={'filter': 'i', 'tract': 1, 'patch': '8,2'}:   File ""src/image/Image.cc"", line 92, in static typename lsst::afw::image::ImageBase<PixelT>::_view_t lsst::afw: :image::ImageBase<PixelT>::_makeSubView(const lsst::afw::geom::Extent2I&, const lsst::afw::geom::Extent2I&, cons t typename lsst::afw::image::detail::types_traits<PixelT, false>::view_t&) [with PixelT = double]     Box2I(Point2I(-2,2),Extent2I(11,11)) doesn't fit in image 7x15 {0} lsst::pex::exceptions::LengthError: 'Box2I(Point2I(-2,2),Extent2I(11,11)) doesn't fit in image 7x15' Traceback (most recent call last):   File ""/sps/lsst/Library/lsstsw/stack/Linux64/pipe_base/2015_10.0-3-g24e103a/python/lsst/pipe/base/cmdLineTask.py"", line 321, in __call__     result = task.run(dataRef, **kwargs)   File ""/sps/lsst/dev/lsstprod/clusters/my_packages/pipe_tasks/python/lsst/pipe/tasks/multiBand.py"", line 552, in run     self.deblend.run(exposure, sources, exposure.getPsf())   File ""/sps/lsst/Library/lsstsw/stack/Linux64/pipe_base/2015_10.0-3-g24e103a/python/lsst/pipe/base/timer.py"", line 118, in wrapper     res = func(self, *args, **keyArgs)   File ""/sps/lsst/Library/lsstsw/stack/Linux64/meas_deblender/2015_10.0-3-ga5a97e7/python/lsst/meas/deblender/deblend.py"", line 231, in run     self.deblend(exposure, sources, psf)   File ""/sps/lsst/Library/lsstsw/stack/Linux64/pipe_base/2015_10.0-3-g24e103a/python/lsst/pipe/base/timer.py"", line 118, in wrapper     res = func(self, *args, **keyArgs)   File ""/sps/lsst/Library/lsstsw/stack/Linux64/meas_deblender/2015_10.0-3-ga5a97e7/python/lsst/meas/deblender/deblend.py"", line 308, in deblend     clipStrayFluxFraction=self.config.clipStrayFluxFraction,   File ""/sps/lsst/Library/lsstsw/stack/Linux64/meas_deblender/2015_10.0-3-ga5a97e7/python/lsst/meas/deblender/baseline.py"", line 354, in deblend     psf, pk, sigma1, patchEdges)   File ""/sps/lsst/Library/lsstsw/stack/Linux64/meas_deblender/2015_10.0-3-ga5a97e7/python/lsst/meas/deblender/baseline.py"", line 1073, in _handle_flux_at_edge     psfim = psfim.Factory(psfim, Sbox, afwImage.PARENT, True)   File ""/sps/lsst/Library/lsstsw/stack/Linux64/afw/2015_10.0-8-g4057726/python/lsst/afw/image/imageLib.py"", line 4630, in Factory     return ImageD(*args)   File ""/sps/lsst/Library/lsstsw/stack/Linux64/afw/2015_10.0-8-g4057726/python/lsst/afw/image/imageLib.py"", line 4472, in __init__     this = _imageLib.new_ImageD(*args) LengthError:   File ""src/image/Image.cc"", line 92, in static typename lsst::afw::image::ImageBase<PixelT>::_view_t lsst::afw::image::ImageBase<PixelT>::_makeSubView(const lsst::afw::geom::Extent2I&, const lsst::afw::geom::Extent2I&, const typename lsst::afw::image::detail::types_traits<PixelT, false>::view_t&) [with PixelT = double]     Box2I(Point2I(-2,2),Extent2I(11,11)) doesn't fit in image 7x15 {0} lsst::pex::exceptions::LengthError: 'Box2I(Point2I(-2,2),Extent2I(11,11)) doesn't fit in image 7x15' {code} The second problem may occur when the overlap region between a PSF image and the data image it corresponds to is only 1 pixel in either dimension. In any case, there's a gap in the graceful-failure logic that could let such a problem through, which would result in received error message: {code:hide-linenum} measureCoaddSources FATAL: Failed on dataId={'filter': 'i', 'tract': 1, 'patch': '8,1'}: Traceback (most recent call last):   File ""/sps/lsst/Library/lsstsw/stack/Linux64/pipe_base/2015_10.0-3-g24e103a/python/lsst/pipe/base/cmdLineTask. py"", line 321, in __call__     result = task.run(dataRef, **kwargs)   File ""/sps/lsst/dev/lsstprod/clusters/my_packages/pipe_tasks/python/lsst/pipe/tasks/multiBand.py"", line 552, i n run     self.deblend.run(exposure, sources, exposure.getPsf())   File ""/sps/lsst/Library/lsstsw/stack/Linux64/pipe_base/2015_10.0-3-g24e103a/python/lsst/pipe/base/timer.py"", l ine 118, in wrapper     res = func(self, *args, **keyArgs)   File ""/sps/lsst/Library/lsstsw/stack/Linux64/meas_deblender/2015_10.0-3-ga5a97e7/python/lsst/meas/deblender/de blend.py"", line 231, in run     self.deblend(exposure, sources, psf)   File ""/sps/lsst/Library/lsstsw/stack/Linux64/pipe_base/2015_10.0-3-g24e103a/python/lsst/pipe/base/timer.py"", l ine 118, in wrapper     res = func(self, *args, **keyArgs)   File ""/sps/lsst/Library/lsstsw/stack/Linux64/meas_deblender/2015_10.0-3-ga5a97e7/python/lsst/meas/deblender/de blend.py"", line 308, in deblend     clipStrayFluxFraction=self.config.clipStrayFluxFraction,   File ""/sps/lsst/Library/lsstsw/stack/Linux64/meas_deblender/2015_10.0-3-ga5a97e7/python/lsst/meas/deblender/ba seline.py"", line 312, in deblend     tinyFootprintSize=tinyFootprintSize)   File ""/sps/lsst/Library/lsstsw/stack/Linux64/meas_deblender/2015_10.0-3-ga5a97e7/python/lsst/meas/deblender/ba seline.py"", line 575, in _fitPsfs     **kwargs)   File ""/sps/lsst/Library/lsstsw/stack/Linux64/meas_deblender/2015_10.0-3-ga5a97e7/python/lsst/meas/deblender/ba seline.py"", line 752, in _fitPsf     sx1, sx2, sx3, sx4 = _overlap(xlo, xhi, px0+1, px1-1)   File ""/sps/lsst/Library/lsstsw/stack/Linux64/meas_deblender/2015_10.0-3-ga5a97e7/python/lsst/meas/deblender/ba seline.py"", line 721, in _overlap     (xlo <= xhi)  and (xmin <= xmax)) AssertionError {code}",2 DM-4744,"Demonstrate web authentication using CILogon and Globus","Configure mod_auth_oidc on lsst-auth1 with CILogon and Globus.",4 DM-4745,"CONOPS to support design activities.","One foundational document required for how NCSA Astronomy Core Services working methods that is missing is a concept of operations (CONOPS) for the L1 system. Better late than never, I wrote a L1 cops illustrating the uses of the L1 system, allowing for further specifications and context for all staff involved in the project. the document currently exists a a draft in google docs. It's been review by Mario and KT. This is more properly a systems engineering document, not at all sure where a final home for it belongs (or how it acquires status)",10 DM-4746,"Prepare activity diagrams and backing conops for LI provisioning and ARP, including satellite computing centers.","Prepared two longer con accompanied by (hand drawn activity diagrams). One conops /activity diagram describe the work at the archive center to provide and support the L1 services used by telescope operations. the second activity diagram and conops respdes to Chuck Clavers' request to have materials that explain the relationship of the stiletto computing center at CCIN2P3 to Archive Center at NCSA. Both are DRAFT; and coops seem to be systems engineering documents, and where to deliver a blessed version and who is responsible for this is unclear to me. ",5 DM-4747,"Beth Willman visit","self explanatory",2 DM-4748,"Review gartner materials relating ITIL, Devops and related topic","Read Gartner materials related to ITIL, devops IT organization in preparation for more detailed thinking about Data Operations. One major category of thought is ""Mode 1 and Mode 2"" type organizations. Mode ! is the current typical controlled environment the strength is when something precious needs to be managed. For LSST this might be the data release production, which is baselined to be 9 months on a unique resource that the project procured. Mode 2 is ""doves"" which is best used for nimble, fall fast software. An example e testing algorithms.",3 DM-4749,"Management for December","includes attendance at the NSF CI for facilities workshop (2d). Hiring, hiring related presentation at U of I ACI. Management of group and effort distributed at NCSA. Oversight of some legacy projects (Chileand data center, etc)",9 DM-4751,"Connecting table with histogram viewer","Create a demo, which takes a URL and shows a table with a histogram viewer connected to it.",6 DM-4753,"Cleanup location of anonymous namespaces","we place anonymous namespace in two ways: (a), INSIDE lsst::qserv::<module> namespace, or (b) BEFORE. This story involves cleaning it up - move them to before lsst::qserv::<module>",1 DM-4754,"Add mysql connection to QueryContext ","We need access to database schema for various reasons (analyzing queries, checking authorization, for queries like ""show create table"" and others).",3 DM-4755,"Implement globally unique queryId",NULL,7 DM-4756,"Support human-friendly Thread ID in logging messages","Per discussion 1/6/2016, it'd be nice to have a function that generates user-friendly threadId on Linux to simplify debugging.",4 DM-4758,"create multi image viewer","* port of MultiDataView.java * support grids, rows, finger chart type grid * support paging with table data sets * support DatasetInfoConverter port * The components should be able to display any group of data * Critical for Firefly Viewer",12 DM-4759,"Port Data set info converter achitechture","defines various image data types, how to get them, groupings, artifacts. I am not quite happy with how we did in in GWT so the design needs to be improved. Must be less complex.",8 DM-4760,"L1 Concept of Operations (December work)","Assist in developing a ConOps for the L1 system. This is the first step to making a detailed design and plan for construction. - Revised/cleaned up/added to L1 ConOps - Meeting to clarify calibration products production use case requirements - Discussions about operational use cases, processes, functions of L1 system",10 DM-4761,"Ops Planning - December","- LOPT and TOWG meetings - Beth Willman 2-day visit; discussions about operations, proposal timeline and deliverables - Prepared FTE estimates for IT roles - Reviewed ITIL roles and clarified work descriptions - Met with NCSA ICI leads to get input on FTE estimates for various roles - Timing diagrams - Worked on first draft of cycle diagram showing 24 hours of operations at NCSA",7 DM-4762,"Margaret's mgmt. activities in December","- Meetings: security, IdM, DMLT, supertask coordination, standups, etc. - Staffing - ARI meeting and preparation - Reviewed resumes, discussed staffing plan - TPR, invoice breakouts, milestones - Discussed EV process and MIS tool design for internal management - etc. ",13 DM-4763,"Exploration of In-memory database packages used in time critical applications","Begin evaluation of potential in-memory data storage tools - selecting memcached and redis to start. - 4 With the intent to gain familiarity with these tools, procurred introductory volume on redis and began writing prototype python code to prototype lists, hashes, and lists of hashes. Sketched out and implemented base python class with virtual save method, then wrote child classes for replicators, replicator health, replicator jobs etc. and tested this code and implemented the save methods. - 4 Installed the above code on a nebula instance that acts as a job manager, then ran real job messages through the system and simulated task assignment and completion, using redis to track jobs. - 2 Exploring how a logging and visualization harness might be included so job activity could 1) be observed in realtime, and 2) so a session could be played back as an after action review to investigate errors, bottlenecks, cold restart behavior, etc. - 2 To finish out this story, redis replication must be included in the above prototype.",12 DM-4764,CONOPS-V1,"Evaluated first cut of CONOPS document. Formulated queries for clarifying specific questions regarding ldm-230 - 2",2 DM-4765,"Track and provide feedback on base site facitlity",NULL,2 DM-4766,"Build network testbed",NULL,4 DM-4767,"add CSS import and image import and clean up some existing jsx",NULL,2 DM-4768,"Port W16 CModel improvements from HSC","Three significant changes were made to CModel in [HSC-1339|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1339]. They were described by [~jbosch] in a [post to {{hsc_software}}|http://jeeves.astro.princeton.edu/pipermail/hsc_software/all/4568.html]. They include: * Changing the method by which the initial approximation is determined; * Changing the determination of the pixel region to use in fitting; * A new prior on ellipticity and radius. Please port these changes to LSST. Also include the results of fixing the bug described in [HSC-1384|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1384].",4 DM-4771,"Week end 12/12/15","Support for lsst-dev cluster, OpenStack, and accounts for week ending December 12, 2015.",2 DM-4772,"Week end 12/19/15","Support for lsst-dev cluster, OpenStack, and accounts for week ending December 19, 2015.",2 DM-4773,"New equipment setup and configuration (week end 12/05/15)","* Setup IPMI on lsst-esxi3, lsst-esxi5, lsst-esxi6 ** Created ipmitools binary - Ubuntu 12.04 has the correct libraries for ESXi6 ** Installed on lsst-esxi3, lsst-esxi5, lsst-esxi6 ** Configuration worked well on lsst-esxi5, lsst-esxi6. ** lsst-esxi3 is actually lsst-esxi4. System will not respond to ipmitool commands - suggest waiting for scheduled outage and manually setting up ipmi. * Debugged networking issues with new equipment * Cleanup of crashplan archives for new lsst system * Mac VMs ** Working on figuring out Mac VM requirements and process with Josh Hobblitt ** Attempted to install and configure puppet on Mac VMs - running into configuration issues ",3 DM-4774,"New equipment setup and configuration (week end 12/12/15)","* Set up IPMI on lsst-test1, lsst-test2,…lsst-test9 * More research on using Puppet on Mac VMs – little progress * Cleanup of NFS space in ITS ",2 DM-4775,"New equipment setup and configuration (week end 12/19/15)","* Research on using puppet on Mac VMs ** Considering using Vagrant to manage VirtualBox or Fusion Mac VMs ** Tried to setup LDAP auth for Mac user auth * Cleanup of NFS space in ITS",2 DM-4777,"Updates to the Sizing Model","Updated processor projections based upon Haswell and Skylake expectations. Added Shipping rates, Chilean and US power and cooling rates, updated memory pricing projections. Working with Spectra Logic on updating and improving tape predictions, library space and power requirements, upgrade options and mapping of bandwidth and capacity requirements to hardware (need to figure in fudge factors for latency of mounting tapes, latency of seeks times, maybe space for tape migrations, replace replacement tapes with updating pricing that includes tape replacement). Inclusion of that into the document will be in the next story.",3 DM-4778,"Contractual work, justifications, inventory for LSST hardware","Reviewing hardware purchase contracts, reviewing internal hardware budget justifications (and attending related meetings), working with purchasing on 'vendor specific' purchasing options, incorporating updated vendor-quoted pricing into expected hardware expenditures. ",2 DM-4780,"meas_extensions_shapeHSM seems to be broken","I have installed the meas_extensions_shapeHSM package together with galsim and tmv (I documented it at : https://github.com/DarkEnergyScienceCollaboration/ReprocessingTaskForce/wiki/Installing-the-LSST-DM-stack-and-the-related-packages#installing-meas_extensions_shapehsm) and tried to run it on CFHT cluster data. My config file is the following: {code:python} import lsst.meas.extensions.shapeHSM config.measurement.plugins.names |= [""ext_shapeHSM_HsmShapeRegauss"", ""ext_shapeHSM_HsmMoments"", ""ext_shapeHSM_HsmPsfMoments""] config.measurement.plugins['ext_shapeHSM_HsmShapeRegauss'].deblendNChild='' config.measurement.slots.shape = ""ext_shapeHSM_HsmMoments"" {code} When I run measCoaddSources.py, I get the following error : {code} Traceback (most recent call last): File ""/sps/lsst/Library/lsstsw/stack/Linux64/pipe_tasks/2015_10.0-10-g1170fd0/bin/measureCoaddSources.py"", line 3, in <module> MeasureMergedCoaddSourcesTask.parseAndRun() File ""/sps/lsst/Library/lsstsw/stack/Linux64/pipe_base/2015_10.0-3-g24e103a/python/lsst/pipe/base/cmdLineTask.py"", line 444, in parseAndRun resultList = taskRunner.run(parsedCmd) File ""/sps/lsst/Library/lsstsw/stack/Linux64/pipe_base/2015_10.0-3-g24e103a/python/lsst/pipe/base/cmdLineTask.py"", line 192, in run if self.precall(parsedCmd): File ""/sps/lsst/Library/lsstsw/stack/Linux64/pipe_base/2015_10.0-3-g24e103a/python/lsst/pipe/base/cmdLineTask.py"", line 279, in precall task = self.makeTask(parsedCmd=parsedCmd) File ""/sps/lsst/Library/lsstsw/stack/Linux64/pipe_base/2015_10.0-3-g24e103a/python/lsst/pipe/base/cmdLineTask.py"", line 363, in makeTask return self.TaskClass(config=self.config, log=self.log, butler=butler) File ""/sps/lsst/Library/lsstsw/stack/Linux64/pipe_tasks/2015_10.0-10-g1170fd0/python/lsst/pipe/tasks/multiBand.py"", line 530, in __init__ self.makeSubtask(""measurement"", schema=self.schema, algMetadata=self.algMetadata) File ""/sps/lsst/Library/lsstsw/stack/Linux64/pipe_base/2015_10.0-3-g24e103a/python/lsst/pipe/base/task.py"", line 255, in makeSubtask subtask = configurableField.apply(name=name, parentTask=self, **keyArgs) File ""/sps/lsst/Library/lsstsw/stack/Linux64/pex_config/2015_10.0-1-gc006da1/python/lsst/pex/config/configurableField.py"", line 77, in apply return self.target(*args, config=self.value, **kw) File ""/sps/lsst/dev/lsstprod/clusters/my_packages/meas_base/python/lsst/meas/base/sfm.py"", line 247, in __init__ self.initializePlugins(schema=self.schema) File ""/sps/lsst/dev/lsstprod/clusters/my_packages/meas_base/python/lsst/meas/base/baseMeasurement.py"", line 298, in initializePlugins self.plugins[name] = PluginClass(config, name, metadata=self.algMetadata, **kwds) File ""/sps/lsst/dev/lsstprod/clusters/my_packages/meas_base/python/lsst/meas/base/wrappers.py"", line 15, in __init__ self.cpp = self.factory(config, name, schema, metadata) File ""/sps/lsst/dev/lsstprod/clusters/my_packages/meas_base/python/lsst/meas/base/wrappers.py"", line 223, in factory return AlgClass(config.makeControl(), name, schema) File ""/sps/lsst/dev/lsstprod/clusters/my_packages/meas_extensions_shapeHSM/python/lsst/meas/extensions/shapeHSM/hsmLib.py"", line 964, in __init__ def __init__(self, *args, **kwargs): raise AttributeError(""No constructor defined - class is abstract"") AttributeError: No constructor defined - class is abstract {code}",1 DM-4781,"MariaDB does not work together with mysql-proxy","We have switched to MAriaDB but there is one issue that complicates things - mysql client from mariadb fails to connect to mysql-proxy with an error: {noformat} ERROR 1043 (08S01): Bad handshake {noformat} so Fabrice had to find a workaround for our setup to use client from mysqlclient package instead. This workaround is not perfect and it complicates other things. Would be nice to make things work transparently for mariadb. ",2 DM-4782,"JIRA project for the publication board","The LSST Publication Board requests a JIRA project for managing its workload. ",2 DM-4783,"Rename temporarily mariadb client","MariaDB client isn't compliant with mysqlproxy and eups doesn't allow to override it with regular mysql client, so it will be rename, so that MYSQLCLIENT_DIR reference in qserv table file can be removed (indeed, it brokes qserv_distrib setup, but not qserv setup, ...)",1 DM-4784,"Consulting in December",NULL,3 DM-4785,"Update provenance in baseline schema","Current provenance schema in baseline (cat/sql) is very old and no longer reflect latest thinking. This story involves bringing cat/sql up to data and replacing existing prv_* tables with tables we came up with in the epic.",2 DM-4786,"Packge mysqlproxy 0.8.5","See https://mariadb.atlassian.net/browse/MDEV-9389",2 DM-4788,"FITS Visualizer porting: Mouse Readout: part 2: flux value","Call the server when mouse pauses, include the flux value in the readout. The should also include support for 3 color.",4 DM-4789,"FITS Visualizer porting: Mouse Readout: part 3: Lock by click & 3 color support","add toggle button that make the mouse readout lock to last position click on. It will not longer update on move but by click Include: 3 Color Support",8 DM-4790,"S17 Refactor MySQL Connection in Qserv",NULL,38 DM-4791,"F17 Setup Qserv and ImgServ with PanSTARRS data","Once the PanSTARRS data becomes public, we should load it to qserv. This epic involves partitioning data, loading to qserv and making it ready for analysis by friendly scientists (and for our internal testing). We should also make the panstarrs images available through imgserv, The work involves setting up webserv instance and configuring imgserv for panstarrs.",24 DM-4793,"Refactor prototype docs into “Developer Guide” and Science Pipelines doc projects","Refactor [lsst_stack_docs|https://github.com/lsst-sqre/lsst_stack_docs] into two doc projects - LSST DM Developer Guide that will be published to {{developer.lsst.io}}, and - LSST Science Pipelines that will be published to {{pipelines.lsst.io}}",3 DM-4794,"Write Zoom Options Popup","Write the simple zoom options popup that is show when the user clicks zoom too fast or the zoom level exceeds the maximum size. activate this popup from visualize/ui/ZoomButton.jsx",2 DM-4798,"DetectCoaddSourcesTask.scaleVariance gets wrong result","DetectCoaddSourcesTask.scaleVariance is used to adjust the variance plane in the coadd to match the observed variance in the image plane (necessary after warping because we've lost variance into covariance). The current implementation produces the wrong scaling in cases where the image has strongly variable variance (e.g., 10 inputs contributed to half the image, but only 1 input contributed to the other half) because it calculates the variance of the image and the mean of the variance separately so that clipping can affect different pixels. Getting this scaling very wrong can make us dig into the dirt when detecting objects, with drastic implications for the resultant catalog. This is a port of [HSC-1357|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1357] and [HSC-1383|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1383].",1 DM-4799,"Rotate Popup",NULL,4 DM-4801,"Update the ground truth values in the lsst_dm_demo to reflect new defaults in deblending","In DM-4410 default configuration options were changed such that footprints are now grown in the detection task, and the deblender is run by default. This breaks the lsst_dm_demo, as now the results of processing are slightly different. The short term solution as part of DM-4410 was to run the demo with the defaults overridden to be what they were prior to DM-4410. In the long term the values used in the compare script should be updated to reflect what would be generated with running processCcd with the stack defaults. ",1 DM-4805,"Some wcs keywords need to be removed from the metadata of raw DECam data","Header keys such as PVi_j left in the raw metadata confuse the making of wcs in later processing steps. For example, when {{calexp}} is read in {{makeDiscreteSkyMap.py}}, {{makeCoaddTempExp.py}}, and so on, this message appears: {code:java} makeWcs: Interpreting RA---TAN-SIP/DEC--TAN-SIP + PVi_j as TPV makeWcs WARNING: Stripping PVi_j keys from projection RA---TPV-SIP/DEC--TPV-SIP {code} These {{calexp}} are created by running {{processCcd.py}} on raw data, and are mis-interpreted as TPV. ",7 DM-4806,"Test stack with mariadbclient","Now that we switched Qserv to mariadb, it'd be good to switch the rest of the stack. This story involves trying out if things still work if we switch mysqlclient to mariadbclient.",2 DM-4807,"Add Shared Scan Table Information to CSS ","Some information should be added to CSS to indicate if a table should be locked in memory for shared scans and the effect the table is likely to have on the time it takes to complete a query.",4 DM-4808,"Package mariadbclient","There are some very low level modules that depend on mysqlclient (for example daf_persistence). It'd be too harsh to make them depend on mariadb, so we should package mariadb client.",1 DM-4809,"X16 Fine-tune Shared Scans","Fine tune shared scans code, in particular take advantage of unique queryId.",29 DM-4810,"Provenance Prototyping","Build a proof-of-concept provenance prototype for a selected pipeline, perhaps HCS.",50 DM-4814,"Create validation_data set for DECam validation test","Create a `validation_data_decam` to provide a few images for DECam validation tests. Use the COSMOS field data as currently available on NCSA being processed by [~nidever]. Select just a few images for now.",2 DM-4815,"Planning for Software Documentation Deployment Service","Write initial draft of [SQR-006|http://sqr-006.lsst.io] that specifies how the documentation deployment service will work.",4 DM-4817,"Read and understand `ci_hsc` and plan relationship with `validate_drp`","Read through and run the `ci_hsc` tests and plan for how this module and efforts should relate to `validate_drp`. a. Add capabilities to `validate_drp` to run the tests in `ci_hsc`. (/) b. Compare frameworks. (/) c. Plan for how such validation and continuous integration data sets should be constructed. (/) ",2 DM-4820,"Improvement of raw data handling in DecamMapper","Two minor improvements with better coding practice: - Be more specific copying FITS header keywords. Avoid potential problems if unwelcome keywords appear in the header in the future. Suggested in the discussions in DM-4133. - Reuse {{isr.getDefectListFromMask}} for converting defects. A more efficient method that uses the FootprintSet constructor with a Mask and a threshold has just been adopted in DM-4800. Processing is not changed effectively. ",1 DM-4821,"HSC backport: Remove interpolated background before detection to reduce junk sources","This is a port of [HSC-1353|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1353] and [HSC-1360|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1360]. Descriptions from HSC: {panel:title=HSC-1353} We typically get a large number of junk detections around bright objects due to noise fluctuations in the elevated background. We can try to reduce the number of junk detections by adding an additional local background subtraction before object detection. We can then add this back in after detection of footprints and peaks. {panel} {panel:title=HSC-1360} I forgot to set the useApprox=True for the background subtraction that runs before footprint and peak detection. This will then use the Chebyshev instead of the spline. {panel}",1 DM-4822,"Code review","DM-4133, DM-4800, DM-4709, DM-4707, DM-4814",4 DM-4823,"Add Dropdowns to Vis toolbar","Add the dropdown to the vis tool bar",2 DM-4824,"Clean up div and css layout on FitsDownloadDialog","FitsDownload dialogs html and css is not quite right. Needs some clean up.",1 DM-4825,"makeDiscreteSkymap has a default dataset of 'raw'","The default dataset type for command line tasks is raw. In the case MakeDiscreteSkyMapTask is asking the butler for calexp images. This shouldn't be a problem, but in my case I have calexp images, but no raw images. This causes the task to think there is no data to work on, so it exits.",1 DM-4826,"Understand async queries in Qserv","Try to understand, without doing actual implementation what is involved in implementation of support for asynchronous queries in Qserv and possibly web interface. Should result in a roadmap for implementation at all levels. ",10 DM-4827,"Adapt `validate_drp` to standard python and bin subdir sturcture","Move Python files into python/lsst namespace convention. Decide on where {{validateCfht.py}} and {{validateDecam.py}} executables should live Add package requirements to {{ups/validate_drp.table}}",1 DM-4829,"Finish Fits View Decoration: context toolbar, title, expand button, etc",NULL,5 DM-4830,"Add error handling to PsfFitter in meas::modelfit","The {{ShapeletPsfApprox}} Task uses a class called {{PsfFitter}} which is not a {{SimpleAlgorithm}} and does not support error handling. Add error handling to this class, and modify the Task definition in {{psf.py}} to call an {{errorHandler fail()}} function when the algorithm's {{optimizer.run()}} call fails. Also, add unit tests",6 DM-4831,"Add bright object masks to pipeline outputs","Given per-patch inputs providing {code} id, B, V, R, ra, dec, radius {code} for each star to be masked, use this information to set: * A bit in the mask plane for each affected pixel * A flag in the source catalogues for each object that has a centroid lying within this mask area This is a port of [HSC-1342|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1342] and [HSC-1381|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1381].",3 DM-4833,"Update configuration for Suprime-Cam","The {{obs_subaru}} configuration for Suprime-Cam needs updating to match recent changes in the stack. Port of [HSC-1372|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1372].",1 DM-4834,"Preliminaries for LSST vs HSC pipeline comparison through coadd processing","This is the equivalent of DM-3942 but through coadd processing. Relevant HSC tickets include: * [HSC-1371|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1371]",1 DM-4835,"Allow slurm to request total CPUs rather than nodes*processors.","On some systems, we are asked to request a total number of tasks, rather than specify a combination of nodes and processors per node. It also makes sense to use the SMP option this way. This is a port of [HSC-1369|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1369].",2 DM-4836,"Fix logic for applying aperture corrections","With the current flow, the aperture corrections are being applied only after all the measurement plugins have run through, independent of their execution order. This results in plugins whose measurements rely on aperture corrected fluxes (i.e. with execution order > APCORR_ORDER) being applied prior to the aperture correction, leading to erroneous results. The only plugin currently affected by this is {{base_ClassificationExtendedness}}. This ticket involves applying a temporary fix to ensure proper application and order of aperture corrections. However, the problem highlights the fact that the current logic of how and when aperture corrections are applied should be reworked (on another ticket) to be less error-prone.",6 DM-4837,"Implement brighter-fatter correction","Please port the prototype Brighter-Fatter correction work by Will Coulton from HSC. This covers [HSC-1189|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1189], [HSC-1332| https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1368], [HSC-1368|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1368]. Note also the stand alone commits [783b124|https://github.com/HyperSuprime-Cam/obs_subaru/commit/783b124b6813f5745ce1e444f61fb0114d055907] and (if this work is performed after DM-3373) [9fc5e78| https://github.com/HyperSuprime-Cam/obs_subaru/commit/9fc5e78247e7173e095255dba34e994f73a6bd1d].",4 DM-4839,"High-level overview of DRP processing","Create high-level overview of Data Release Production, probably as an annotated flowchart, for use in sizing model work and as a graphical table of contents for more detailed descriptions.",6 DM-4840,"Add sky objects","Please add ""sources"" corresponding to empty sky (ie, at positions where nothing else has been detected) and include them in multiband processing. This is a port of [HSC-1336|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1336] and [HSC-1358|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1358]. ",4 DM-4841,"Use high S/N band as reference for multiband forced photometry","We are currently choosing the priority band as the reference band for forced photometry as long as it has a peak in the priority band regardless of the S/N. Please change this to pick the highest S/N band as the reference band when the priority band S/N is sufficiently low. This is a port of [HSC-1349|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1349].",1 DM-4842,"Don't write HeavyFootprints in forced photometry","There's no need to persist {{HeavyFootprint}}s while performing forced photometry since retrieving them is as simple as loading the _meas catalog. This is a port of [HSC-1345|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1345].",1 DM-4847,"Add new blendedness metric","[HSC-1316|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1316] shifts the calculation of blendedness from {{meas_deblender}} to {{meas_algorithms}} and defines a new blendedness metric in the process. Please port it.",3 DM-4848,"Measure photometric repeatability and correctness of reported errors","1. Calculate and plot photometric variability across series of N images. Compare to reported photometric errors. Designed for N > 5. 2. Calculate and plot Delta flux / sigma_flux for multiple observations of stars in field. This is related to 1. but is focused on N=2 to N=5. 3. Fit uncertainty distribution vs. magnitude to identify any floor in the photometric uncertainty and to check performance vs. photon counts.",5 DM-4849,"LDM-151 - comments from Jacek","I am reading your https://github.com/lsst/LDM-151/blob/draft/DM_Applications_Design.tex, and I have some minor comments suggestions. I am going to add comments to this story to capture it. Feel free to apply to ignore :)",1 DM-4850,"Factor out duplicate setIsPrimaryFlag from MeasureMergedCoaddSourcesTask and ProcessCoaddTask","{{MeasureMergedCoaddSourcesTask.setIsPrimaryFlag()}} and {{ProcessCoaddTask.setIsPrimaryFlag()}} are effectively the same code. Please split this out into a separate task which both of the above can call. This is a (partial) port of [HSC-1112|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1112] and should include fixes from [HSC-1297|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1297].",2 DM-4851,"XY Plot action and reducers","Write action and reducers for XY Plot",6 DM-4852,"Implement zenodio.metadata to mediate Zenodo's API with local YAML metadata","[Zenodio|http://zenodio.lsst.io] is a Python package we’re building to interact with Zenodo. For our various doc/technote/publishing projects we want to use YAML files (embedded in a Git repository, for example) to maintain deposition metadata so that the upload process itself can be automated. The {{zenodio.metadata}} sub package provides a Python representation of Zenodo metadata (but not File or Zenodo deposition metadata). See DM-4725 for the upload API work, which consumes the metadata objects.",2 DM-4856,"Add __setitem__ for columns in afw.table","It's confusing to have to use an extra {{[:]}} to set a column in afw.table, and we can make that unnecessary if we override {{\_\_setitem\_\_}} as well as {{\_\_getitem\_\_}}.",2 DM-4857,"Replace killproc and pidofproc with kill and pidof","Running at NCSA on OpenStack revealed that our qserv-stop.sh and qserv-status.sh fail because of missing killproc and pidofproc. It looks like (see eg http://stackoverflow.com/questions/3013866/killproc-and-pidofproc-on-linux) these are not very portable and it is better to use kill and pidof.",1 DM-4858,"imagesDiffer doesn't handle overflow for unsigned integers","I'm seeing a test failure in afw's testTestMethods.py, apparently due to my numpy (1.8.2) treating images that differ by -1 as differing by 65535 in both {{numpy.allclose}} and array subtraction (which doesn't promote to an unsigned type). Does this still cause problems in more recent versions of {{numpy}}? If not, I imagine it's up to me to find a workaround for older versions if I want it fixed? (assigning to [~rowen] for now, just because I know he originally wrote this test and I hope he might know more)",1 DM-4861,"Please provide ""getting started"" documentation on writing meas_base algorithms","{{meas_base}} provides a framework for writing measurement algorithms in a uniform way. However, documentation on exactly how this should be done is fragmentary: * There's some basic documentation in [Doxygen|https://lsst-web.ncsa.illinois.edu/doxygen/x_masterDoxyDoc/meas_base.html] which provides a useful introduction, but doesn't discuss common idioms and helpers such as {{FlagHandler}}, {{SafeCentroidExtractor}} and transformations. * The [design notes on Confluence|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=20284390] are not intended as documentation and aren't kept up-to-date as new features are added, but can still be a useful reference. * [~jbosch] gave a [nice introduction|https://github.com/lsst-dm/Oct15_bootcamp/blob/measurement/measurement/measurement.pdf] at the October 2015 Bootcamp, but a set of slides is no substitute for proper documentation, and again there's no expectation that these will be kept up-to-date. Please provide a centralized, maintained guide to writing {{meas_base}} plugins.",4 DM-4862,"Add point selection","click and highlight a point. Is on when mouse readout ""Lock by Click"" is on. However, can me turned on externally by adding toolbar context menu options.",2 DM-4864,"Setup webserv for SUI","This story involves setting up a webserv in a VM (NCSA OpenStack) with a small data set: images and corresponding database catalog. We need to * setup VM * build the stack for webserv and qserv * identify images to load * run qserv in the VM * run ingest to load the data to mysql (mysql will run on lsst10) and qserv (qserv will run directly in the VM) * run two webservers - one with mysql backend, one with qserv backend * open the port numbers for the IPAC team",10 DM-4865,"Port HSC background matching routines","HSC has its own implementation of background matching: see {{background.py}} in {{hscPipe}}. Please port it to the LSST stack.",4 DM-4866,"Filter mask planes propagated to coadds","Some mask planes -- {{CROSSTALK}}, {{NOT_DEBLENDED}} -- do not need to be propagated to coadds. Add an option to remove them. This is a port of work performed on [HSC-1174|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1174] and [HSC-1294|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1294].",2 DM-4867,"scisql build scripts are buggy ","The scisql build script logic for MySQL/MariaDB version checking is broken on all platforms. There are also assumptions about shared library naming that do not hold on OS/X, which means that the deployment scripts are likely broken on all platforms other than Linux.",2 DM-4868,"Setup LSST stack for verification datasets work","Created a script to setup required LSST stack packages for bulge survey processing. ",10 DM-4869,"Installing a reference catalog to use in bulge survey processing","Install Astrometry.net Index Files for 2MASS all sky catalog",5 DM-4870,"Setup orchestration environment at lsstdev for bulge survey processing","Follow instructions at https://confluence.lsstcorp.org/display/DM/1.+Quick+Start+-+LSST+Cluster+Orchestration to set up the required packages to run the lsst stack at lsst cluster.",10 DM-4871,"Debugging lsst.astrometry task in bulge survey processing","The DECam bulge survey is being processed as part of the verification data sets effort. During astrometry calibration task a large number of failures (affecting ~100 of 213 visits) have been found in calibrate.astrometry.matcher. We report here details of the investigation around this issue. Part of the task is to learn how to use the task built in debug. More info: https://confluence.lsstcorp.org/display/SQRE/Bulge+Survey+Processing#BulgeSurveyProcessing-Results",10 DM-4872,"Setup firefly example for image visualization ","Start from example provided by the firefly team https://github.com/lsst/suit",10 DM-4873,"Test the matchOptimisticB astrometric matcher","The matchOptimisticB matcher fails on many visits of the bulge verification dataset. This prompted a deeper investigation of the performance of the matcher. Angelo and David developed a test script and discovered that the matcher works well with offsets of the two source catalogs of up to 80 arcsec, but fails beyond that. This should be robust enough for nearly all datasets that the LSST stack will be used on.",3 DM-4874,"Write a firefly search processor that retrieves image paths from the butler",NULL,5 DM-4875,"base has no readme","The base package does not have a readme file, so it's unclear what it's for. The package name is also somewhat unfortunate, being so generic, but at least with a readme it would be clearer how important it is (if it is, in fact, important).",1 DM-4876,"Compile list of DM simulation needs for Andy Connolly","Compile list of DM simulation needs over the next ~6 months to give to Andy Connolly (simulations lead).",3 DM-4877,"Diagnostic plot showing the number of process ccd failures in each visit as function of the density of sources.","Diagnostic plot showing the number of process ccd failures in each visit as function of the density of sources. - Use the butler to iterate over the data ids, read the src catalog and count the number of sources per ccd. - Use afw.display.ds9 to display the image and overlay the sources",5 DM-4878,"Propagate flags from individual visit measurements to coadd measurements","It is useful to be able to identify suitable PSF stars from a coadd catalogue. However, the PSF is not determined on the coadd, but from all the inputs. Add a mechanism for propagating flags from the input catalogues to the coadd catalogue indicating stars that were used for measuring the PSF. Make the inclusion fraction threshold configurable so we can tweak it (so we only get stars that were consistently used for the PSF model; the threshold might be set it to 0 for ""or"", 1 for ""all"" and something in between for ""some""). Make the task sufficiently general that it can be used for propagating arbitrary flags. This is a port of work carried out on [HSC-1052|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1052] and (part of) [HSC-1293|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1293].",2 DM-4879,"Make coadd input catalogs contiguous","It's convenient if we can assume that coadd input catalogs are contiguous -- it simplifies the implementation of {{PropagateVisitFlagsTask}} (DM-4878), for example. Make it so. This is port of work carried out on [HSC-1293|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1293].",1 DM-4880,"Test capabilities of python bokeh plotting library for making interactive plots - I","We are testing the bokeh library plot for interactive visualization in the web, the python API is atractive and allows rapid prototype which is good for SQuaRE build up its QA system. Some examples are available in this repo, including a scatter plot with linked histograms in both axis which seems really useful. https://github.com/lsst-sqre/bokeh-plots A more complete demonstration of bokeh is available in this webminar: https://continuum-analytics.wistia.com/medias/f6wp9dam91 ",20 DM-4882,"base_Variance plugin generates errors in lsst_dm_stack_demo","Since DM-4235 was merged, we see a bunch of messages along the lines of: {code} processCcd.measurement WARNING: Error in base_Variance.measure on record 427969358631797076: The center is outside the Footprint of the source record {code} in the output from {{lsst_dm_stack_demo}}. (See e.g. [here|https://ci.lsst.codes/job/stack-os-matrix/label=centos-6/7482/console#console-section-3]). It's not fatal, but the warnings are disconcerting and could be indicative of a deeper problem.",2 DM-4884,"S17 Improve Qserv Integration Tests","Integration tests need improvements, in particular, we want to run multi-node integration tests easily (possibly without docker), get rid of mono-node test. We should catch errors from individual tests.",15 DM-4885,"Improve/simplify multi-node tests",NULL,10 DM-4886,"Script launch of HTCondor pool on Nebula OpenStack","The ability to automate the launch of htcondor pools of worker nodes (e.g., with LSST software installed) on the Nebula OpenStack can be useful in several ways for LSST DM. On one hand, users can start up their own customized pool should the standard pool available on other resources such as lsst-dev not be suitable (e.g., not enough cores, not enough memory per core/slot, customized software not installed on the systems, etc.) Also, scripted launch of a pool can be part of the development of a solution for offering ""batch"" scheduling to the Nebula OpenStack, following an approach similar to e.g., CANFAR ( http://www.canfar.net/docs/batch/ , http://cloudscheduler.org) whereby a ""cloud scheduler"" is used in conjunction with an htcondor central manager to provide batch access to the cloud (i.e., submitted jobs are placed into an htcondor queue, that will execute on launched instances, when those instances join the working pool.) ",24 DM-4887,"Refactor measurement afterburners into a new plugin system","Some of the operations we currently run as part of measurement (or would like to) share some features that make them a bit different from most plugin algorithms: - They must be run after at least some other high-level plugins, and may be run after all of them. - They do not require access to pixel data, as they derive their outputs entirely from other plugins' catalog outputs. - They may require an aggregation stage of some sort to be run on the regular plugin output before they can be run. Some examples include: - Star/Galaxy classification (with training done after measurement and before classification). - Applying aperture corrections (estimating the correction must be done first). - BFD's P, Q, R statistics (requires a prior estimated from deep data). We should move these algorithms to a new plugin system that's run by a new subtask, allowing these plugins to be run entirely separately from {{SingleFrameMeasurementTask}}. This will simplify some of the currently contorted logic required to make S/G classification happen after aperture correction, while making room for hierarchical inference algorithms like BFD and Bayesian S/G classification in the future. (We will not be able to support BFD immediately, as this will also require changes to our parallelization approach, but this will be a step in the right direction). This work should *probably* be delayed until after the HSC merge and [~rowen]'s rewrite of {{ProcessCcdTask}} are complete, but it's conceivable that this refactoring could solve emergent problems there and be worth doing earlier as a result.",8 DM-4888,"Update git-lfs documentation to work with git-lfs 1.1.0+","The git-lfs client (1.1.0+) does not support empty username and passwords. To work around this, users can store the appropriate credentials directly with the credential helper.",3 DM-4889,"Update git-lfs repositories to point to the git-lfs documentation.","Update git-lfs repositories to point to the git-lfs documentation. All documentation should be generic and point to: http://developer.lsst.io/en/latest/tools/git_lfs.html ",1 DM-4892,"Take DECam data with collimated beam projector","[~mfisherlevine] and [~rhl] will travel to CTIO to observe on the Blanco 4m telescope.",30 DM-4893,"Write tutorial describing remote IPython + ds9 on lsst-dev","[~mfisherlevine] recently figured out how to set up his system to run a remote IPython kernel on {{lsst-dev}} and interact with it from his laptop, including streaming image display from the remote system to a local instance of {{ds9}}. He will write all this up so that others in the community can easily do the same.",2 DM-4894,"Ingest DECam/CBP data into LSST stack","[~mfisherlevine] will ingest the data taken in DM-4892 into the LSST stack. Initial experiments indicate problems with: * Bias subtraction * Flat fielding * Bad pixel masks These may already be remedied by work on {{obs_decam}}; if not, he will file stories and fix them.",3 DM-4895,"Prepare calibration products for analysing DECam data","Determine if existing bad pixel masks, flats, etc are adequate for analysing the DM-4893 DECam data, and, if not, provide alternatives.",3 DM-4897,"Qualitative exploration of the CBP/DECam data","Having got the CBP/DECam data loaded into the stack, explore the parameter space and understand data. This should result in a series of stories describing more detailed analysis with quantitative results.",8 DM-4898,"Implement the simulation and testing framework for analyzing image differencing","We need to be able to test all aspects of image differencing. This includes template generation, astrometric registration, and differencing. We know that DCR will be an effect that will need to be mitigated so we will have to be able to simulate it and show how well various techniques deal with it.",50 DM-4899,"Implement simulations for testing image differencing.","Implement a suite of simulations tools for testing the image differencing techniques, specifically with an eye toward dealing with DCR.",15 DM-4901,"Use yaml configuration files to store camera-specific data ID and ref image information for validation testing.","Currently there is {{validateCfht.py}} and {{validateDecam.py}} as code. These differ in just having {{defaultData}} functions that specify the dataIds to consider and the dataIds to use as a reference for comparison. Storing the information necessary to create these sets of dataIds in separate data files, to be stored as YAML would 1. Improve the separation of code and data 2. Clarify the usage and necessary information to run on a new or different set of data 3. Make it easier to run different subsets easily by specifying a different input file The proposals is that {{validateCfht.py}} and {{validateDecam.py}} would disappear from {{bin}} and be replaced by just {{validate_drp.py}}. The examples in {{examples/runCfhtTest.sh}} and {{examples/runDecamTest.sh}} will be updated to show the new usage. The README will also be updated.",1 DM-4903,"Expand button hide/show, delete button hide/show, display title options,","Expand button hide/show, delete button hide/show, display title options, support pv.hideTitleDetail to control showing zoom level and rotation info (used by planck) support external title bar (planck as well) support checkbox on title bar (planck) ",4 DM-4904,"Buffer overrun in wcslib causes stack corruption","The buffer 'msg' in wcsfix.c is used to report attempts by wcslib to re-format units found in fits files. It is allocated on the stack (in function 'unitfix') using a pre-processor macro defined size of 160 chars (set in wcserr.h). When attempting to run the function 'unitfix' in wcsfix, this buffer can overflow on some fits files (the raw files generated by HSC seem particularly prone to triggering this behavior) and results in the session being terminated on Ubuntu 14.04 as stack protection is turned on by default i.e. the stack crashes with a 'stack smashing detected' error. We have reported the bug to the creators of wcslib. As a temporary workaround, users affected by the bug should increase the default size of 'msg' by increasing WCSERR_MSG_LENGTH defined in wcserr.h We are providing a small python example that demonstrates the problem. Run it as python test.py <path to ci_hsc>/raw/<any fits file in this directory> We are also providing a simple c program to demonstrate the bug. Compile it as cc -fsanitize=address -g -I$WCSLIB_DIR/include/wcslib -o test test.c -L$WCSLIB_DIR/lib -lwcs (on Linux) cc -fsanitize=address -g -L$WCSLIB_DIR/lib -lwcs -I$WCSLIB_DIR/include/wcslib -o test test.c (on Mac OS X)",2 DM-4906,"Investigate astrometry warnings from processing raw DECam data","This ticket includes efforts to troubleshoot and improve processing DECam raw data in Jan 2016. Investigations of the warnings from solving astrometry led to DM-4805, DM-4859. This ticket also includes partial efforts in DM-4859. For validation, I ran {{processCcd.py}} with two visits of raw Stripe 82 DECam data with and without the changes in DM-4859 (using the first camera geometry fix, which is different from the final fix). The script {{validateDecam}} in {{validate_drp}} is used to check astrometric scatter of the sources between the two visits. Output plots are in the attachments. By fixing DM-4859, the median astrometric scatter (mag < 21) decreases from 31.3 mas to 26.1 mas. The number of matches between two visits increases from 56768 to 71265. Also attached are the ra/dec patches plots, using the {{showVisitSkyMap.py}} script from DM-4095. The two colors represent the two visits of calexp wcs. Southern CCDs had bad astrometric solutions before DM-4859. The validation results are consistent with the patches visualization. ",12 DM-4907,"Cyber security infrastructure requirements","Documenting cyber security operational requirements by LSST, particularly at the obs. site.",2 DM-4908,"Security plan renewal","Continuing work on cyber sec. plan renewal. DM moving along, PO slated next.",1 DM-4909,"DM security meeting","Security meeting/planning with LSST DM team at NCSA.",1 DM-4911,"LSST IaM meetings at NCSA",NULL,1 DM-4912,"LSST IaM bi-weekly coordinating meeting","Meeting between NCSA CSD group, NCSA LSST DM group, and other LSST groups.",1 DM-4915,"Add option for object name resolution","For some object names resolved by NED, the position is not right. In this situation, it would be better to get the position from Simbad. Currently, Firefly offers two options: first NED, then Simbad; first Simbad, then NED. The third option to be added would be ""the best position according to the object type"". It should check the object type returned by NED, making a decision whether to get position from Simbad instead; and vice versa. ",6 DM-4916,"Test obs_decam with processed data","Sometimes DECam-specific bugs only reveal in or affect the processed data. For example the bug of DM-4859 reveals in the {{postISRCCD}} products. If the bugs are DECam-specific, some changes in {{obs_decam}} are likely needed. It would be useful to have a more convenient way to test those changes. In this ticket I modify {{testdata_decam}} so that those data can be processed, and then allow wider options in the {{obs_decam}} unit tests. I add {{testProcessCcd.py}} in {{obs_decam}} that runs {{processCcd.py}} with raw and calibration data in {{testdata_decam}}. Besides a short sanity check, I add a test (testWcsPostIsr) that tests DM-4859. {{testWcsPostIsr}} fails without the DM-4859 fix, and passes with it. ",3 DM-4917,"Porting encodeURL of the java FitsDownlaodDialog code to javascript ","When download an image, the proper name needs to be resolved based on the URL and the information about the image. In Java code, it has the following three methods: {code} encodeUrl makeFileName makeTitleFileName {code} These method should be ported to javascript. Thus, the javascript version of the FitsDownloadDialog will save the file in the same manner. ",2 DM-4919,"Test performance of vertical-partition joins in mysql","We are planning to vertically partition some tables (for example Object). We should make sure such joins across say 5, 10 or 20 tables are not a problem for mysql from performance standpoint. The testing involves creating a wide table (say 200 columns) and testing a speed of full scan, then slicing that table vertically into different number of columns and using join to assemble the pieces together.",4 DM-4920,"Support Multi image fits and controls","add toolbar: next, prev arrow buttons, title when multi image fits has image specific titles or title would be cube number. Make sure the store will support multi images, add next, prev actions, etc",4 DM-4921,"Make obs_subaru build with OS X SIP","Because of OS X SIP, {{obs_subaru}} fails to build on os x 10.11. In the {{hsc/SConscript}} file, the library environment variables need properly set, and scripts need to be delayed until the shebang rewriting occurs. ",1 DM-4923,"want to see locations in trace when butler raises because multiple locations were found","daf_persistence 11.0-2-g56eb0a1+1 gives the unhelpful error message: {code} > RuntimeError: Unable to retrieve bias for {'category': 'A', 'taiObs': '2015-12-22', 'visit': 7292, 'site': 'S', 'dateObs': '2015-12-22', 'filter': 'PFS-M', 'field': 'DARK', 'spectrograph': 2, 'ccd': 5}: No unique lookup for ['calibDate', 'calibVersion'] from {'category': 'A', 'taiObs': '2015-12-22', 'visit': 7292, 'site': 'S', 'dateObs': '2015-12-22', 'filter': 'PFS-M', 'field': 'DARK', 'spectrograph': 2, 'ccd': 5}: 2 matches {code} (the old butler did this too). The user wants to know what the 2 matches were -- it's user error, but the user needs help and printing the first few options (nicely formatted) is very useful. I think I did this on the HSC side. The butler code in question is actually in butlerUtils/python/lsst/daf/butlerUtils/mapping.py and my post-doc gave me the wrong package. It's in need(): {code} > if len(lookups) != 1: > raise RuntimeError, ""No unique lookup for %s from %s: %d matches"" % (newProps, newId, len(lookups)) {code}",1 DM-4925,"Make FlagHandler, SafeCentroidExtractor usable from Python","The {{meas_base}} framework includes {{SafeCentroidExtractor}}, a convenience routine for extracting a centroid from a source record, setting a consistent set of flags if that's not possible or if the centroid is in some way compromised. This consistent flag handling is made possible by the use of the {{FlagHandler}} class. Unfortunately, {{FlagHandler}} is not meaningfully usable from Python, not least because it's impossible to define flags: {code:python} >>> import lsst.meas.base as measBase >>> measBase.FlagDefinition(""flag"", ""doc"") [...] TypeError: __init__() takes exactly 1 argument (3 given) >>> fd = measBase.FlagDefinition() >>> fd.name = ""flag"" [...] AttributeError: You cannot add attributes to <lsst.meas.base.baseLib.FlagDefinition; proxy of <Swig Object of type 'lsst::meas::base::FlagDefinition *' at 0x10a82b900> > {code} Looking further, even were we able to create {{FlagDefinitions}}, the {{FlagHandler}} is initialized with pointers to the beginning/end of a container of them, which seems like a stretch for Python code. Please add Python support for these routines.",2 DM-4926,"Centroids fall outside Footprints","In DM-4882, we observed a number of centroids measured while running the {{lsst_dm_stack_demo}} routines fall outside their associated {{Footprints}}. This was seen with both the {{NaiveCentroid}} and the {{SdssCentroid}} centroiders. For the purposes of DM-4882 we quieted the warnings arising from this, but we should investigate why this is happening and, if necessary, weed out small {{Footprints}} entirely.",8 DM-4928,"Fix intermittent testQdisp failure","The mocks used in the executive class don't mock cancellation correctly and doing so would require significant effort. When Executive::squash() is called, the mocks threads are already running but waiting on the _go barrier. squash() calls JobQuery::cancel() for each thread and cancel() calls markComplete() for the job because a QueryResource has not been aquirred from xrootd. Once all the jobs are cancelled and _go is set to true, the ex.join() command doesn't wait for the jobs to complete since markComplete() has already been called for all of the jobs. If any of the jobs take longer to complete than the main thread, they call markComplete for an Executive that no longer exists and cause the test to fail.",1 DM-4929,"Fix build of MariaDB on OS X El Capitan","The current MariaDB EUPS package does not build on OS X El Capitan because OS X no longer ships with OpenSSL developer files. MariaDB has a build option to use a bundled SSL library in preference to OpenSSL but the logic for automatically switching to this version breaks when the Anaconda OpenSSL libraries are present.",1 DM-4930,"Deploy 4 bare metal hosts for testing Base to Archive transfer implementation","James needs to test network communication methodologies in an environment that mimics the expected real-world conditions. In order to minimize the complications with debugging, using bare metal machines in the first phase is preferred. We can use 4 of the machines bought off the 2015 purchase or purchase new machines just for this purpose.",4 DM-4931,"Qserv build fails on El Capitan with missing OpenSSL","Qserv does not build on OS X El Capitan due to the absence of OpenSSL include files. Apple now only ship the OpenSSL library (for backwards compatibility reasons). Qserv only uses SSL in two places to calculate digests (MD5 and SHA). This functionality is available in the Apple CommonCrypto library. Qserv digest code needs to be taught how to use CommonCrypto.",2 DM-4932,"Track kernel panic issue","The line that caused the kernel panic is in modules/mysql/MySqlConnection.cc line 151. Currently the line is fine and is: std::string const killSql = ""KILL QUERY "" + std::to_string(threadId); This version of the line will occasionally cause the kernel panic (note the missing %1% that should be after KILL QUERY). std::string killSql = boost::str(bo ost::format(""KILL QUERY "") % threadId); ",3 DM-4933,"Create a utility function do do spherical geometry averaging","I would like to calculate a correct average and RMS for a set of RA, Dec positions. Neither [~jbosch] nor [~price] knew of an easy, simple function to do that that existed in the stack. [~price] suggested: {code} mean = sum(afwGeom.Extent3D(coord.toVector()) for coord in coordList, afwGeom.Point3D(0, 0, 0)) mean /= len(coordList) mean = afwCoord.IcrsCoord(mean) {code} That makes sense, but it's a bit unobvious (it's obvious how it works, but would likely never occur to someone that they should do it that way in the stack). Pedantically it's also not the best way to do a mean while preserving precision, but I don't anticipate that to be an issue in practice. Creating a function that did this would provide clarity. I don't know where that function should live. Note: I know how to do this in Astropy. I'm intentionally not using astropy here. But part of the astropy dependency discussion is likely ""how much are we otherwise rewriting in the LSST stack"".",1 DM-4934,"on-going support to Camera team in visualization at UIUC","Attend the weekly meeting and answer questions as needed",2 DM-4936,"Enable validateMatches in ci_hsc","{{python/lsst/ci/hsc/validate.py}} in {{ci_hsc}} [says|https://github.com/lsst/ci_hsc/blob/69c7a62f675b8fb4164065d2c8c1621e296e40ad/python/lsst/ci/hsc/validate.py#L78]: {code:python} def validateMatches(self, dataId): # XXX lsst.meas.astrom.readMatches is gone! return {code} {{readMatches}} (or its successor) should be back in place as of DM-3633. Please enable this test.",2 DM-4937,"multiple CVEs relevant to mariadb 10.1.9 and mysql","Multiple CVEs have been released this week for mysql & mariadb. The current eups product for mariadb is bundling 10.1.9, which is affected. Several of the CVEs do not yet provide details, which typically means they are ""really bad"". https://github.com/lsst/mariadb/blob/master/upstream/mariadb-10.1.9.tar.gz https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-0505 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-0546 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-0596 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-0597 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-0598 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-0600 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-0606 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-0608 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-0609 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-0616 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-2047",1 DM-4938,"Update scisql to v0.3.5","In order to update MariaDB to v10.1.10 {{scisql}} needs to also be updated to deal with the hard-coded version checking. For the current version we get this error with the latest MariaDB: {code} ::::: [2016-01-28T16:51:40.539306Z] user_function(self) ::::: [2016-01-28T16:51:40.539334Z] File ""/home/build0/lsstsw/build/scisql/wscript"", line 63, in configure ::::: [2016-01-28T16:51:40.539346Z] ctx.check_mysql() ::::: [2016-01-28T16:51:40.539392Z] File ""/home/build0/lsstsw/build/scisql/.waf-1.6.11-30618c54883417962c38f5d395f83584/waflib/Configure.py"", line 221, in fun ::::: [2016-01-28T16:51:40.539410Z] return f(*k,**kw) ::::: [2016-01-28T16:51:40.539432Z] File ""tools/mysql_waf.py"", line 85, in check_mysql ::::: [2016-01-28T16:51:40.539451Z] (ok, msg) = mysqlversion.check(version) ::::: [2016-01-28T16:51:40.539473Z] File ""tools/mysqlversion.py"", line 74, in check ::::: [2016-01-28T16:51:40.539514Z] if not comparison_op(version_nums, constraint_nums): ::::: [2016-01-28T16:51:40.539547Z] UnboundLocalError: local variable 'constraint_nums' referenced before assignment Failed during rebuild of DM stack. {code}",1 DM-4939,"IRSA developer mentoring effort","IRSA is contributing to the Firefly package development. we need to put in time to mentor the developers. ",2 DM-4940,"IRSA developer mentoring effort","IRSA is contributing to Firefly development. We need to mentor the new developers.",2 DM-4942,"Fix type inference and return types makeMaskedImage et al","The {{makeMaskedImage}} function and cousins like {{makeExposure}} don't do the type inference they're supposed to do in C++, because they use the old {{typename Image<T>::Ptr}} approach instead of {{PTR(Image<T>)}}. They also return raw pointers, which is dangerous. They should be converted to return shared_ptrs. Note that this will have to include adjusting or removing Swig code (probably {{%newobject}} statements) that deal with taking ownership of the raw pointers.",1 DM-4943,"Switch to MemManReal in the worker scheduler","First iteration of worker scheduler uses a skeleton of the memory manager that doesn't actually look at any of the tables, files, or memory. The scheduler needs to be switched to MemManReal. ",15 DM-4944,"butler should transparently allow files to be compressed or not","see the conversation on c.l.o. at [https://community.lsst.org/t/how-does-the-butler-support-compression/502]. The summary is, when the mapper returns e.g. a non compressed file name e.g. {{foo.fits}}, that file may be compressed and the filename may reflect this e.g. in reality it might be named {{foo.fits.gz}}. On a posix system some component of the butler framework should discover this and transform the filename to the correct filename and pass that to the deserializer. TBD if the list of allowed extensions is hard coded someplace (in a mapper subclass?) or specified another way, perhaps by the policy (could be for dataset type or globally).",6 DM-4945,"Schemas for QA information","This ticket is to capture preliminary design work we are doing for storage of QA system information, which we are working with the Database team on. As well as prior experience, Jacek has made us aware of the sdqa tables in the schema: https://lsst-web.ncsa.illinois.edu/schema/index.php?sVer=baseline and also plan on mining pipeQA for quantities of interest. Once we have a draft, there will be an RFD for soliciting further input. ",10 DM-4946,"afw Wcs object copying does not copy exactly","A probable bug in WCSLIB is causing {{wcscopy}} to create copies of {{Wcs}} objects which are not the same as the object that was copied. In some cases when this object is passed to {{wcsset}} it fails, as the {{Wcs}} object contains impossible values. This has behaviour is non-deterministic (failure is only seen occasionally). The error has only been observed on OSX, but we do not believe it to be operating system dependent (except insofar as different systems and compilers produce different memory layouts and hence different failure modes). This reliably causes {{ci_hsc}} to fail when running on a Mac. Relevant lines in {{afw}} are {{image/Wcs.cc:140}} and the {{Wcs}} copy constructor in {{image/Wcs.cc:468}} Additionally a bug has been found in {{image/Wcs.cc}} on line 485 where the flag property should be set on an element, and not on the object itself, ie {{_wcsInfo[i]->flag = -1;}}.",6 DM-4948,"Please improve the documentation for TransformTask and derivatives","While working on DM-4629 (overhauling {{ProcessCcdTask}}) [~rowen] stumbled over {{TransformTask}}, which he wasn't previously familiar with. Existing Doxygen documentation covers what this task does, but lacks context as to why it's useful. Please provide a high-level overview of what the intention is here.",2 DM-4949,"Improve MySQL proxy code and add unit tests","QServ's proxy needs some cleanup: 1. Standardize passing of q and qU parameters to methods 2. Comment removal may have never worked 3. Whitespace translation is likely buggy 4. A few unit tests verifying routing of queries would be nice ",4 DM-4950,"Build MVP of ltd-keeper web app covering ltd-mason interface","This ticket is to create an MVP of the ltd-keeper web app (RESTful API) that tracks versions of LSST the Docs’ published software documentation. Specifically this ticket will implement the RESTful endpoints needed by ltd-mason. See [SQR-006|http://sqr-006.lsst.io] for design information. [SQR-006|http://sqr-006.lsst.io] will be updated in this ticket as the design is clarified in implementation.",14 DM-4951,"Add S3/Route53 project provisioning capabilities to ltd-keeper","An **authenticated** user should be able to provision (and likewise, delete) an entire published software documentation project via ltd-keeper’s RESTful API. This includes creating an S3 bucket in SQuaRE’s AWS account and setting up Route 53 DNS. The user should also be able to delete a project. This ticket will add AWS affordances to ltd-keeper. DM-4950 will be responsible for hooking this functionality into the methods that service API calls.",3 DM-4952,"delegate argument parsing to CmdLineTask instances","Command-line argument parsing of data IDs for {{CmdLineTask}} s is currently defined at the class level, which means that we cannot make data ID definitions dependent on task configuration. That in turn requires custom {{processCcd}} scripts for cameras that start processing at a level other than ""raw"" (SDSS, DECam with community pipeline ISR, possibly CFHT). Instead, we should let {{CmdLineTask}} *instances* setup command-line parsing; after a {{CmdLineTask}} is constructed, it will have access to its final configuration tree, and can better choose how to parse its ID arguments. I've assigned this to Process Middleware for now, since that's where it lives in the codebase, but it may make more sense to give this to [~rowen], [~price], or [~jbosch], just because we've already got enough familiarity with the code in question that we could do it quickly. I'll leave that up to [~swinbank], [~krughoff], and [~mgelman2] to decide.",2 DM-4955,"Update pyfits","The final version of {{pyfits}} has just been released. This ticket covers updating to that version. This will be helpful in determining whether the migration to {{astropy.io.fits}} will be straightforward or complicated.",1 DM-4956,"Adapt SRD-based measurements of astrometric performance for validate_drp","Adapt the SRD-based specifications for calculation of astrometric performance. Follow the examples for AM1, AM2 as presented at https://confluence.lsstcorp.org/pages/viewpage.action?pageId=41785659 and detailed in DM-3057, DM-3064",4 DM-4957,"Generate JSON output from validate_drp for inclusion in a test harness","Generate JSON output from validate_drp for inclusion in a test harness. Generate a file that summarizes the key metrics calculated by `validate_drp`. Develop naming conventions that will make it easy to plug into the eventual harness being developed as part of DM-2050.",2 DM-4958,"Hard copy support- saving regions","This ticket will only do the region saving. The scope has change somewhat since region saving will talk a little longer and making the png requires some server side work. DM-6139",10 DM-4959,"ci_hsc fails to execute tasks from with SCons on OSX 10.11/SIP","The {{ci_hsc}} package executes a number of command line tasks directly from SCons based on {{Command}} directives in a {{SConstruct}} file. On an OSX 10.11 system with SIP enabled, there are two distinct problems which prevent the necessary environment being propagated to the tasks: * -The {{scons}} executable starts with a {{#!/usr/bin/env python}}. Running through {{/usr/bin/env}} strips {{DYLD_LIBRARY_PATH}} from the environment.- (duplicates DM-4954) * SCons executes command using the [{{sh}} shell on posix systems|https://bitbucket.org/scons/scons/src/09e1f0326b7678d1248dab88b28b456fd7d6fb54/src/engine/SCons/Platform/posix.py?at=default&fileviewer=file-view-default#posix.py-105]. By default, that means {{/bin/sh}} on a Mac, which, again, will strip {{DYLD_LIBRARY_PATH}}. Please make it possible to run {{ci_hsc}} on such a system.",1 DM-4960,"LSST vs. HSC stack comparison: PSF estimation","In order to determine the cause of the output differences between single frame processing runs of the same data using the LSST vs. HSC stacks (see figures attached to DM-4730), a detailed look at some of the image characterization steps is required. This ticket involves a detailed investigation of the initial PSF estimation including: {panel: title=LSST vs. HSC stack runs:} - a comparison of the initial object detection (will likely involve looking at the initial background estimate as well as the specific assignment of footprints) - which objects are selected as PSF candidates - the initial PSF model (as a function of position) {panel}",5 DM-4961,"Obs_Subaru camera mapper has wrong deep_assembleCoadd_config","When lsst switched to using SafeClipAssembleCoaddTask, the camera mapper for hsc was not updated accordingly. This causes ci_hsc to fail when it attempts to verify the config class type for the deep_coadd. Camera mapper should be updated accordingly",1 DM-4962,"January Operation Support Related Tasks","Account cleanup process for existing infrastructure (Identify accounts, assign sponsors) Reconcile inventory between NCSA and Aura (on going). Mock request was generated by Aura for dry run audit. Several machines have been found not included in inventory. Task to be completed in February.",3 DM-4963,"Investigate Roger as fallover for Nebula","Investigate Roger OpenStack as fallback for Nebula during outages. Internally, this required technical and coordination meetings. Externally, this required interfacing with Square in order to facilitate a proper evaluation. This task is ongoing. ",2 DM-4964,"January AAA Tasks","Attended local AAA meetings and reviewed documentation. ",1 DM-4965,"January Tasks","Security meeting with Paul, Bill, Eyrich to review goals and coordinate efforts. Initial draft of the procurement plan. Waiting for hardware contract. Updates to internal FY16 cost estimate spreadsheet and planning (not LDM-144). Updating expected expenditures based on update quotes. Meetings and discussions with OBFS with respect to new vendors within MHEC and procurement approval processing for FY16 components.",4 DM-4966,"January Tasks","Technology and pricing updates to LDM-144. Explanation update in LDM-143. Meetings with multiple vendors re: longer term technology forecasts.",15 DM-4967,"January Tasks","Mtg w/ IN2P3 re: ITIL implementation experiences. Mtg w/ IN2P3 re: tape recall ordering",1 DM-4968,"Jason January Tasks","Activities this month include: IT sys admin meetings, LSST internal project meetings, conducting, coordinating, discussing interviews. Meeting with candidates. ICI coordination meeting (Randy). Discussion of work-to-be-done with onboarded teammates. Relaying task prioritization to IT for LSST-related activities. ",4 DM-4969,"DM Power Requirements","Further discussions about power requirements at the Chilean DC.",2 DM-4970,"Investigate logging, monitoring and metrics technologies and architecture","Investigate technologies and architectures to use with panopticon, our logging system. Perform preliminary research and evaluations into ELK (Elasticsearch, Logstash and Kibana), extensions to ELK and other alternatives.",24 DM-4971,"Meetings, Jan 2016","verfication dataset meetings, TechTalk, RFD, local middleware-related meetings, etc",2 DM-4972,"LOE, Jan 2016","LSST local group meetings, postdoc meeting, other local meetings, etc",2 DM-4973,"Reconsider high detection threshold in CharacterizeImageTask","[~price] makes the [reasonable recommendation|https://community.lsst.org/t/why-was-detection-includethresholdmultiplier-10-for-old-processccdtask/500/6] that we consider providing PSF estimation with the N brightest sources in the image, rather than only detecting bright sources. [~rowen] reasonably believes that this is beyond the scope of DM-4692, hence this new issue. There should be very little new code needed here, but it may involve quite a bit of experimentation and validation.",8 DM-4983,"upstream patches/deps from conda-lsst","Where ever possible, missing dep information and patches from conda-lsst should be upstreamed. The patches have already been observed to cause builds to fail due to upstream changes.",3 DM-4985,"Finish data distribution prototype (March)",NULL,3 DM-4990,"Prepare for auth session at JTM","Prepare for JTM session with a working title of “How Authentication/Authorization technology can be used to implement and enforce data access rights and operational processes for LSST"". Prepare a final title and agenda for the session. Tuesday from 3:30pm - 5:00pm.",5 DM-4991,"Save algorithm metadata in multiband.py","The various {{Tasks}} in {{multiband.py}} do not attach the {{self.algMetadata}} instance attribute to their output tables before writing them out, so we aren't actually saving information like which radii were used for apertures. We should also make sure this feature is maintained in the processCcd.py rewrite.",3 DM-4992,"work flow of light curve visulizaiton","Generate a description document of work flow that a scientist would go through in order to do time series research, visualize the light curve.",4 DM-4993,"review of dependency on the third party packages","We need to periodically review the status of the third party software packages that Firefly depends on. Making a plan to do upgrade if needed. package.json lists out the dependencies Firefly has on the third party software. The attached file was last modified 2016-02-09. package.json_version lists the current version of the third party packages, major changes were indicated by (M). The attached file was created on 2016-02-29. bq. ""babel"" : ""5.8.34"", 6.5.2 (M) ""history"" : ""1.17.0"", 2.0.0 (M) ""icepick"" : ""0.2.0"", 1.1.0 (M) ""react-highcharts"": ""5.0.6"", 7.0.0 (M) ""react-redux"": ""3.1.2"", 4.4.0 (M) ""react-split-pane"": ""0.1.22"", 2.0.1 (M) ""redux-thunk"": ""0.1.0"", 1.0.3 (M) ""redux-logger"": ""1.0.9"", 2.6.1 (M) ""validator"" : ""4.5.0"", 5.1.0 (M) ""chai"": ""^2.3.0"", 3.5.0 (M) ""esprima-fb"": ""^14001.1.0-dev-harmony-fb"", 15001.1001.0-dev-harmony-fb (M) ""babel-eslint"" : ""^4.1.3"", 5.0.0 (M) ""babel-loader"" : ""^5.3.2"", 6.2.4 (M) ""babel-plugin-react-transform"": ""^1.1.0"", 2.0.0 (M) ""babel-runtime"" : ""^5.8.20"", 6.6.0 (M) ""eslint"" : ""^1.10.3"", 2.2.0 (M) ""eslint-config-airbnb"": ""0.1.0"", 6.0.2 (M) works with eslint 2.2.0 ""eslint-plugin-react"": ""^3.5.1"", 4.1.0 (M) works with eslint 2.2.0 ""extract-text-webpack-plugin"": ""^0.8.0"", 1.0.1 (M) ""html-webpack-plugin"": ""^1.6.1"", 2.9.0 (M) ""karma-sinon-chai"": ""^0.3.0"", 1.2.0 (M) ""redux-devtools"" : ""^2.1.2"", 3.3.1 (M) ""webpack"": ""^1.8.2"" 1.12.14, 2.1.0 beta4 (M) ",2 DM-4994,"Design single-sign-on authentication system for webserv","Outline a design of the authentication system (based on components provided by NCSA) that will support single sign-on. Current thinking involves two tokens: application token to certify the app is legitimate and to determine which users it can represent, and user token",6 DM-4995,"Extend webserv API to pass security tokens","Extend the [API|https://confluence.lsstcorp.org/display/DM/AP] to pass security tokens.",8 DM-4996,"Update validate_drp for El Capitan","validate_drp does not work on El Capitan due to SIP (System Integrity Protection) stripping DYLD_LIBRARY_PATH from shell scripts. The simple fix is to add {code} export DYLD_LIBRARY_PATH=${LSST_LIBRARY_PATH} {code} near the top of the scripts.",1 DM-4997,"Benchmark dipole measurement (dipole fitting)","Benchmark dipole measurement (dipole fitting), compare speed directly to psf-fit (and/or galaxy measurement) task. Runtime should be comparable (~factor of two?) - if not understand why. Evaluate new implementation vs. current impl. Accuracy?",8 DM-4998,"Fix rotation for isr in obs_subaru","Approximately half of the HSC CCDs are rotated 180 deg with respect to the others. Two others have 90 deg rotations and another two have 270 deg rotations (see [HSC CCD layout|http://www.naoj.org/Observing/Instruments/HSC/CCDPosition_20150804.png]) . The raw images for the rotated CCDs thus need to be rotated to match the rotation of their associated calibration frames prior to applying the corrections. This is accomplished by rotating the exposure using the *rotated* context manager function in {{obs_subaru}}'s *isr.py* and the *nQuarter* specification in the policy file for each CCD. Currently, *rotated* uses {{afw}}'s *rotateImageBy90* (which apparently rotates in a counter-clockwise direction) to rotated the exposure by 4 - nQuarter turns. This turns out to be the wrong rotation for the odd nQuarter CCDs as shown here: !ccd100_nQuarter3.png|width=200! top left = raw exposure as read in top right = flatfield exposure as read in bottom left = _incorrectly_ rotated raw exposure prior to flatfield correction",2 DM-4999,"Implement new dipole fitting algorithm as SimpleAlgorithm","Implement new dipole fitting algorithm as SimpleAlgorithm -- implement measure, fail methods, define flags",14 DM-5002,"Make ci_hsc resumable","if ci_hsc fails for any reason, (or is cancelled) it must start from the beginning of processing again. This is because of the use of functools.partial to generate dynamic function. These differ enough in their byte code that scons thinks each build has a new function definition passed to the env.command function. Using lambda would suffer from the same problem. This ticket should change how the function signature is calculated such that scons can be resumed. This work does not prevent this from being used as a ci tool, as the .scons directory can be deleted which will force the whole SConstruct file to run again.",2 DM-5005,"Please trim config overrides in validate_drp","validate_drp will test more of our code if it uses default config parameters wherever possible. To that effect I would like to ask you to eliminate all config overrides that are not essential and document the reasons for the remaining overrides. For DECam there are no overrides that are different than the defaults, so the file can simply be emptied (for now). For CFHT there are many overrides that are different, and an important question is whether the overrides in this package are better for CFHT data than the overrides in obs_cfht; if so, please move them to obs_cfht. As a heads up: the default star selector is changing from ""secondMoment"" to ""objectSize"" in DM-4692 and I hope to allow that in validate_drp, since it works better and is better supported. Sorry for the incorrect component, but validate_drp is not yet a supported component in JIRA (see DM-5004)",1 DM-5006,"remove REUSE_DATAREPO in testCoadds in pipe_tasks","When the test fails and the output directory is written but not populated, subsequent test executions fail every time until the output directory is deleted or REUSE_DATAREPO is set to False. This is misleading for users who don't know about this hidden feature. Furthermore, the REUSE_DATAREPO=False feature is broken; setting it False causes NameError: global name 'DATAREPO_ROOT' is not defined. It would be better if the test cleaned up after itself (deleted all outputs) every time. If it's really important to reuse the outputs then the dir should be cleaned up in the case of failed writes and/or corruption. ",1 DM-5008,"F16 Data Access Model Refresh","A refresh of the storage / IO model (LDM-141). The work involves understanding cost impact, and discussing the impact on science. ",33 DM-5011,"Resolve development issues by testing using WAN Emulator","A test plan draft was written and some short meetings were held regarding the use of the WAN Emulator. The manuals for the Apposite Netropy 40G emulator were retrieved and read. The test plan draft for three test projects is attached.",5 DM-5012,"Continued WBS planning","Finished out all necessary fields in the first cut at the WBS. Split project into work phases, Began drilling down into the milestones for each phase, with accurate estimation the goal. Provided miscellaneous diagrams to capture expected functionality for various state transitions throughout the system.",10 DM-5013,"Convert Confluence DM Developer Guide to Sphinx (hack day) ","This is a hack day sprint to convert all remaining content on https://confluence.lsstcorp.org/display/LDMDG to reStructuredText content in the Sphinx project at https://github.com/lsst-sqre/dm_dev_guide and published at http://developer.lsst.io. The top priority for this sprint is to port all content into reST and have it tracked by Git. h2. Sprint ground rules # Before the sprint, clone {{https://github.com/lsst-sqre/dm_dev_guide.git}} and {{pip install -r requirements.txt}} in a Python 2.7 environment so that you can locally build the docs ({{make html}}). # Claim a page from the list below by putting your name on it. Put a checkmark on the page when you’ve merged it to the ticket branch (see below). # See http://developer.lsst.io/en/latest/docs/rst_styleguide.html for guidance on writing our style of reStructuredText. Pay attention to the [heading hierarchy|http://developer.lsst.io/en/latest/docs/rst_styleguide.html#sections] and [labelling for internal links|http://developer.lsst.io/en/latest/docs/rst_styleguide.html#internal-links-to-labels]. # If you use Pandoc to do an initial content conversion, you still need to go through the content line-by-line to standardize the reStructuredText. I personally recommend copy-and-pasting-and-formatting instead of using Pandoc. # Your Git commit messages should include the URL of the original content from Confluence. # Merge your work onto the {{tickets/DM-5013}} ticket branch. Rebase your personal work branch before merging. JSick is responsible for merging this ticket branch to {{master}}. # Put a note at the top of the confluence page with the new URL; root is {{http://developer.lsst.io/en/latest/}}. h2. Planned Developer Guide Table of Contents We’re improving the organization of DM’s Developer Guide; there isn’t a 1:1 mapping of Confluence pages to developer.lsst.io pages. Below is a proposed section organization and page structure. These sections can still be refactored based on discussion during the hack day. h3. Getting Started — /getting-started/ * ✅ *Onboarding Checklist* (Confluence: [Getting Started in DM|https://confluence.lsstcorp.org/display/LDMDG/Getting+Started+in+DM]). I’d like this to eventually be a quick checklist of things a new developer should do. It should be both a list of accounts the dev needs to have created, and a list of important developer guide pages to read next. The NCSA-specific material should be spun out. [[~jsick]] * *Communication Tools* (new + DM Confluence [Communication and Links|https://confluence.lsstcorp.org/display/DM/Communication+and+Links]). I see this as being an overview of what methods DM uses to communicate, and what method should be chosen for any circumstance. * *Finding Code on GitHub* (new). This should point out all of the GitHub organizations that a developer might come across (DM and LSST-wide), and point out important repositories within each organization. Replaces the confluence page [LSST Code Repositories|https://confluence.lsstcorp.org/display/LDMDG/LSST+Code+Repositories] h3. Processes — /processes/ * ✅ *Team Culture and Conduct Standards* (confluence) * ✅ *DM Development Workflow with Git, GitHub, JIRA and Jenkins* (new & Confluence: [git development guidelines for LSST|https://confluence.lsstcorp.org/display/LDMDG/git+development+guidelines+for+LSST] + [Git Commit Best Practices|https://confluence.lsstcorp.org/display/LDMDG/Git+Commit+Best+Practices] + [DM Branching Policy|https://confluence.lsstcorp.org/display/LDMDG/DM+Branching+Policy]) * ✅ *Discussion and Decision Making Process* (new & [confluence|https://confluence.lsstcorp.org/display/LDMDG/Discussion+and+Decision+Making+Process]) * ✅ *DM Wiki Use* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/DM+Wiki+Use]) [[~swinbank]] * ✅ *Policy on Updating Doxygen* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Policy+on+Updating+Doxygen]); needs to be addressed with TCT. Inter-link with the developer workflow page. [[~jsick]] (we’re just re-pointing the Confluence page to the workflow document) * ✅ *Transferring Code Between Packages* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Transferring+Code+Between+Packages]) [[~swinbank]] * -*Policy on Changing a Baseline Requirement*- ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Policy+on+Changing+a+Baseline+Requirement]) * ✅ *Project Planning for Software Development* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Project+Planning+for+Software+Development]) [[~swinbank]] * ✅ *JIRA Agile Usage* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/JIRA+Agile+Usage]) [[~swinbank]] * -*Technical/Control Account Manager Guide*- ([confluence|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=21397653]) (Do not port; see discussion below.) * *Licensing* (new) Need a centralized page to discuss license and copyright policies; include boilerplate statements. h3. Coding Guides — /coding/ * ✅ *Introduction* and note on stringency language (confluence: [DM Coding Style Policy|https://confluence.lsstcorp.org/display/LDMDG/DM+Coding+Style+Policy]) * ✅ *DM Python Style Guide* (confluence: [Python Coding Standard|https://confluence.lsstcorp.org/display/LDMDG/Python+Coding+Standard]) * ✅ *DM C++ Style Guide* (confluence pages: [C++ Coding Standard|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908666] + [C++ General Recommendations|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908756] + [C++ Naming Conventions|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908685] + [C++ Files|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908674] + [C++ Statements|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908706] + [C++ Layout and Comments|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908737] + [Policy on use of C++11/14|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=20283399] + [On Using ‘Using’|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=20283856]) * Coding Style Linters (new; draft from confluence [C++ Coding Standards Compliance|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=20283861] and [Python Coding Standards Compliance|https://confluence.lsstcorp.org/display/LDMDG/Python+Coding+Standards+Compliance] * ✅ *Using C++ Templates* ([confluence|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=20284190]); this page needs to severely edited or re-written, however. * ✅ *Profiling* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Profiling|]). Also add a section ‘Using Valgrind with Python' (new) [[~jsick]] * ✅ *Boost Usage* ([TRAC|https://dev.lsstcorp.org/trac/wiki/TCT/BoostUsageProposal]) [[~tjenness]] * ✅ *Software Unit Test Policy* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Software+Unit+Test+Policy]) [[~swinbank]] * ✅ *Unit Test Coverage Analysis* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Coverage+Analysis]) [[~swinbank]] * ✅ *Unit Testing Private C++ Functions* ([trac|https://dev.lsstcorp.org/trac/wiki/UnitTestingPrivateFunctions]) [[~swinbank]] h3. Writing Docs — /docs/ * *Introduction* (new): Overview of DM’s documentation needs; links resources on technical writing. * *English Style Guide* (new): Supplement the [LSST Style Manual|https://www.lsstcorp.org/docushare/dsweb/Get/Document-13016/LSSTStyleManual.pdf] and provide English style guidance specific to DM. Capitalization of different heading levels; use of Chicago Manual of Style; a ‘this, not that’ table of spelling and word choices. * ✅ *ReStructuredText Style Guide* (new) * ✅ *Documenting Stack Packages* (new) * ✅ *Documenting Python Code* (new) * ✅ *Documenting C++ Code* (confluence, adapted from [Documentation Standards|https://confluence.lsstcorp.org/display/LDMDG/Documentation+Standards]); needs improvement * ✅ *Writing Technotes* (new; port README from [lsst-technote-bootstrap|https://github.com/lsst-sqre/lsst-technote-bootstrap/blob/master/README.rst]) h3. Developer Tools — /tools/ * ✅ *Git Setup and Best Practices* (new) * ✅ *Using Git Large File Storage (LFS) for Data Repositories* (new) * ✅ *JIRA Work Management Recipes* (new) * ✅ *Emacs Configuration* ([Confluence|https://confluence.lsstcorp.org/display/LDMDG/Emacs+Support+for+LSST+Development]). See DM-5045 for issue with Emacs config repo - [~jsick] * ✅ *Vim Configuration* ([Confluence|https://confluence.lsstcorp.org/display/LDMDG/Config+for+VIM]) - [~jsick] h3. Developer Services — /services/ * ✅ *NCSA Nebula OpenStack Guide* (Confluence: [User Guide|https://confluence.lsstcorp.org/display/LDMDG/NCSA+Nebula+OpenStack+User+Guide] + [Starting an Instance|https://confluence.lsstcorp.org/display/LDMDG/Introduction+to+Starting+a+Nebula+Instance] + [Using Snapshots|https://confluence.lsstcorp.org/display/LDMDG/Start+an+Instance+using+a+base+snapshot+with+the+LSST+Stack]. Add the [Vagrant instructions too from SQR-002|http://sqr-002.lsst.io]? [[~jsick]] * ✅ *Using lsst-dev* (Confluence: [notes Getting Started|https://confluence.lsstcorp.org/display/LDMDG/Getting+Started+in+DM] + [Developer Tools at NCSA|https://confluence.lsstcorp.org/display/LDMDG/Developer+Tools+at+NCSA] * ✅ *Using the Bulk Transfer Server at NCSA* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Using+the+Bulk+Transfer+Server+at+NCSA]) [[~jsick]] h3. Build, Test, Release — /build-ci/ * *Eups for LSST Developers* (new) [[~swinbank]] * ✅ *The LSST Software Build Tool* → ‘Using lsstsw and lsst-build' ([confluence|https://confluence.lsstcorp.org/display/LDMDG/The+LSST+Software+Build+Tool]); lsstsw and lsst-build documentation. [[~swinbank]] * *Using DM’s Jenkins for Continuous Integration* (new) [~frossie] * ✅ *Adding a New Package to the Build*([confluence|https://confluence.lsstcorp.org/display/LDMDG/Adding+a+new+package+to+the+build]) [[~swinbank]] * ✅ *Distributing Third-Party Packages with Eups* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Distributing+third-party+packages+with+EUPS]) [[~swinbank]] * ✅ *Triggering a Buildbot Build* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Triggering+a+Buildbot+Build]) [~frossie] * ✅ *Buildbot Errors FAQ* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Buildbot+FAQ+on+Errors]) [~frossie] * * Buildbot configuration ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Buildbot+Configuration+and+Setup] [~frossie] * *Creating a new DM Stack Release* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Creating+a+new+DM+Stack+Release]); though this page or a modern equivalent should probably belong with the software docs? [~frossie] _A lot of work should go into this section._ Have something about Scons? Or maybe that belongs in the doc of each relevant software product. h2. Leftover Confluence pages h3. The following pages should be moved to a separate Confluence space run by NCSA: * [NCSA Nebula OpenStack Issues|https://confluence.lsstcorp.org/display/LDMDG/NCSA+Nebula+OpenStack+Issues] * [DM System Announcements|https://confluence.lsstcorp.org/display/LDMDG/DM+System+Announcements] * [NCSA Development Servers|https://confluence.lsstcorp.org/display/LDMDG/DM+Development+Servers] h3. The following pages are either not relevant, generally misplaced, or need to be updated/recalibrated: * [Git Crash Course|https://confluence.lsstcorp.org/display/LDMDG/Git+Crash+Course] * [Basic Git Operations|https://confluence.lsstcorp.org/display/LDMDG/Basic+Git+Operations] * [Handling Git Push Problems|https://confluence.lsstcorp.org/display/LDMDG/Handling+Git+Push+Problems] * [LSST Code Repositories|https://confluence.lsstcorp.org/display/LDMDG/LSST+Code+Repositories]; see the proposed “Finding Code on GitHub” page for a replacement. * [Standards and Policies|https://confluence.lsstcorp.org/display/LDMDG/Standards+and+Policies]: this is a good TOC for the Confluence docs; but not longer needed for the new docs. * [Documentation Guidelines|https://confluence.lsstcorp.org/display/LDMDG/Documentation+Guidelines]. Some of this could be re-purposed into an intro to the ‘Writing Documentation’ section; some of this should go in a ‘Processes' page. * [DM Acknowledgements of Use|https://confluence.lsstcorp.org/display/LDMDG/DM+Acknowledgements+of+Use]: this probably belongs in documentation for the software projects that actually used this work.",5 DM-5014,"Set doRenorm default to False in AssembleCcdTask","Change the default value of {{AssembleCcdConfig.doRenorm}} to {{False}} for the reasons given in RFC-157 and to implement that RFC.",1 DM-5015,"Optionally report do-nothing config overrides","As discussion on DM-4692 and in various HipChat rooms, it's too easy for camera-level config override files to contain many options that don't actually change anything, because they simply override the defaults with the same default values. To aid in tracking these down and removing them, we should have an option in which {{CmdLineTask}} s (delegating to {{pex_config}}) refuse or warn about overrides that have no effect. We should probably not make failing on do-nothing overrides the default behavior, but we could consider making warning the default behavior. Mostly, I think it's important just to be able to find such options when wanted.",4 DM-5018,"Modernize version check scripts in matplotlib and numpy packages","The version check scripts in the stub {{matplotlib}} and {{numpy}} eups packages use old Python conventions. They should be updated to work with 2.7+.",1 DM-5019,"FITS Visualizer porting: Expanded mode single - part 2","I split DM-4497 into two part so I can demonstrate code reviews. This part has paging controller & layout cleaned up. This tickets is messy because it involves a lot of refactoring of the reducers. Therefore I am going to end it and move the rest of the UI work to DM-5088.",4 DM-5022,"Modernize python code in Qserv scons package","The {{site_scons}} Python code is not using current project standards. For example, print is not a function, exceptions are not caught {{as e}}, {{map}} is called without storing the result and {{map/filter/lambda}} are used where list comprehensions would be clearer. Most of these fixes are trivial with {{futurize}}.",1 DM-5023,"Adds, Moves, Change support for DNS, network, IP addressing, etc",NULL,6 DM-5024,"define rack, pdu specifications and obtain pricing quotes",NULL,3 DM-5025,"Base site and summit RFP","Working with Ron to create a RFP for the acquisition of network equipment for the summit and base. ",10 DM-5026,"Fix dependencies for eups-packaged sqlalchemy","Eups-packaged sqlalchemy lists {{mysqlclient}} as required dependency which is not really right. sqlalchemy does not directly depend on mysql client stuff, instead it determines at run time which python modules it needs to load depending on what exact driver client code is requesting (and {{mysqlclient}} does not actually provides python module so this dependency does not even make anything useful). So dependency on specific external package should be declared on client side and not in sqlalchemy, {{mysqlclient}} should be removed from sqlalchemy.table.",1 DM-5027,"eval NCSA vSphere/OSX support -- first attempt",NULL,1 DM-5028,"Design interconnect for GPFS cluster prototype",NULL,4 DM-5029,"Create physical and logical network diagrams for first phase of purchases",NULL,6 DM-5030,"Tests fail on Qserv on OS X El Capitan because of SIP","OS X El Capitan introduced System Integrity Protection which leads to dangerous environment variables being stripped when executing trusted binaries. Since {{scons}} is launched using {{/usr/bin/env}} the tests that run do not get to see {{DYLD_LIBRARY_PATH}}. This causes them to fail. The same fix that was applied to {{sconsUtils}} (copying the path information from {{LSST_LIBRARY_PATH}}) needs to be applied to the test execution code used by Qserv's private {{site_scons}} utility code.",2 DM-5033,"X16 Data Access and Database Documentation","Update the documentation for Data Access and Database - bring it up to date with the design. This includes LDM-135 (Database Design), and creating a new LDM document or DAX Design).",73 DM-5035,"DAX & DB Docs (Fritz, March)","* Document Data Distribution * Create structure for DAX doc * Bring over Provenance documentation from prov_prototype * Update LDM-135 to reflect the updates to the storage/IO model * Update LDM-152 * Fix LDM-135: 3.3.6.4 and 3.3.6.5 should be 3rd level, so 3.3.7 and 3.3.8 ",3 DM-5036,"DAX & DB Docs (John)","* Refresh shared scans design documentation (in LDM-135) * Add info about query cancellation (in LDM-135)",10 DM-5037,"DAX & DB Docs (AndyS)","* Document db and table metadata * Document async queries * Document data loader",5 DM-5038,"DAX & DB Docs (Nate)","* Improve butler documentation",8 DM-5039,"DAX & DB Docs (Brian)","* Document webserv/imgserv/metaserv/dbserv",10 DM-5040,"DAX & DB Docs (Mike)","Document secondary index",5 DM-5041,"DAX & DB Docs (Serge)","* Document spatial indexing * Document database ingest * Refresh ""Stored Procedures and Function"" in LDM-135",6 DM-5042,"Load panstarrs data to qserv",NULL,19 DM-5043,"Setup webserv with panstarrs data",NULL,5 DM-5049,"update ""newinstall.sh"" nebula images & docker containers","[~hchiang2] is looking for nebula images newer than {{w_2015_45}} (from the exploratory work in DM-4326) and [~gdaues] is interested in images with a complete {{lsst_distrib}} install for orchestration testing. New builds should incorporate the pending change to {{newinstall.sh}} that converts from {{anaconda}} to {{miniconda}}.",6 DM-5050,"SingleFrameVariancePlugin takes variance of entire image","{{SingleFrameVariancePlugin}} takes the median variance of the entire image, rather than within an aperture around the source of interest. A {{Footprint}} is constructed with the aperture, but it is unused. This means that this plugin takes an excessive amount of run time (255/400 sec in a recent run of processCcd on HSC {{visit=1248 ccd=49}} with DM-4692).",1 DM-5052,"Design replacement for A.net index files","We need a simple way to hold index files that will be easy to use and simple to set up.",2 DM-5053,"Some small things slipped through in winter 2016","Fix up things that slipped through or were delayed in winter 2016. The individual things are small parts of larger epics and typically are the result of emergent work or increased scope.",30 DM-5054,"Implement Approx/Interp improvements","We are making due with the current approximation and interpolation scheme, but the two should be merged. This must really be done after the HSC merge because of the difficulty of doing large refactoring before then.",30 DM-5055,"Assess priority of Aprox/Interp upgrades.","This is to assess the priority of a major approximation and interpolation refactor.",4 DM-5056,"RFC corrections for ISR.","Create a list of ISR requirements and have it RFCd.",9 DM-5057,"Assess the corrections that need to be imlemented","The stack can do many of the corrections needed. Assess the status of the current algorithms and identify any deficiencies.",4 DM-5058,"Improve and implement crosstalk in ISR","-ISR needs the following correction algorithms: fringe, crosstalk, overscan (with discontinuity), and non-linearity. Several of these are implemented in HSC and have been ported, but may need some work.- As noted below this will now only be the crosstalk portion of the ISR upgrades.",10 DM-5059,"Design the refactoring for ProcessCcd","There is a significant design issue when refactoring a piece of this importance. Carry out a design study to implement in DM-4692.",12 DM-5068,"Week end 1/09/16","Support for lsst-dev cluster, OpenStack, and accounts for week ending January 9, 2016.",1 DM-5069,"Week end 1/16/16","Support for lsst-dev cluster, OpenStack, and accounts for week ending January 16, 2016.",2 DM-5070,"Week end 1/23/16","Support for lsst-dev cluster, OpenStack, and accounts for week ending January 23, 2016.",2 DM-5071,"Week end 1/30/16","Support for lsst-dev cluster, OpenStack, and accounts for week ending January 23, 2016.",2 DM-5072,"New equipment setup and configuration (week end 1/23/16)","* Finished setting up Mac vSphere infrastructure with Paul",2 DM-5073,"New equipment setup and configuration (week end 1/30/16)","* Set up new lsst-dev7 as CentOS 7 server * Continuing to set up IPMI on new test servers (working with Dell on issue with iDRAC license upgrade)",3 DM-5074,"Decommissioning old equipment (week end 1/16/16)","* Recovery of old LSST used equipment ** Moved remaining surplussed last servers to wiping bench ** Started wiping drives ** Re-purposed 10 Dell 1950 ",2 DM-5075,"Decommissioning old equipment (week end 1/23/16)","* Complete the cleanup of last used NCSA systems",1 DM-5077,"Lenovo test server","* Mount Lenovo test server in LSST1 rack. Install fiber card and networking. Test PXE boot to 10G nic. * Work on getting Lenovo to PXE boot to 10G card * Booted satisfactorily to 1GB interface – loaded Centos 7 ** Abruptly ends after Menu with 10GB card ",3 DM-5078,"PcaPsf can hit an assertion failure","This is bad for multiple reasons: 1. When multiprocessing, the assertion failure kills a single process, which prevents the final join of the multiple processes, so the job hangs forever. 2. The failure is not logged. 3. Hard assertions like this should only occur when we break the system integrity, which this does not (i.e., it's too big a hammer for the problem). {code} pprice@tiger-sumire:/tigress/pprice/dm-4692 $ eups list -s afw tickets.DM-4692-gd8ad35cd96+1 b1901 setup afwdata 2016_01.0 b1901 b1902 setup astrometry_net 0.50.lsst2+5 b1901 b1902 setup astrometry_net_data sdss-dr9-fink-v5b setup base 2016_01.0 b1901 b1902 setup boost 1.59.lsst5 b1901 b1902 setup cfitsio 3360.lsst4 b1901 b1902 setup coadd_chisquared 2016_01.0+6 b1901 setup coadd_utils 2016_01.0+6 b1901 setup daf_base 2016_01.0 b1901 b1902 setup daf_butlerUtils tickets.DM-4692-g048b33c50e+3 b1901 setup daf_persistence 2016_01.0-1-gf47bb69+1 b1901 b1902 setup display_ds9 2015_10.0+43 b1901 setup doxygen 1.8.5.lsst1 b1901 b1902 setup eigen 3.2.5 b1901 b1902 setup fftw 3.3.4.lsst2 b1901 b1902 setup geom 10.0+50 b1901 b1902 setup gsl 1.16.lsst3 b1901 b1902 setup ip_diffim tickets.DM-4692-g543ea8fde5+3 b1901 setup ip_isr 2016_01.0+6 b1901 setup lsst_build LOCAL:/tigress/pprice/lsstsw/lsst_build setup mariadbclient master-gf2dee38289 b1901 b1902 setup matplotlib 0.0.1+5 b1901 b1902 setup meas_algorithms tickets.DM-4692-g3d073a93d7+1 b1901 setup meas_astrom tickets.DM-4692-gbbf15418e6+1 b1901 setup meas_base LOCAL:/tigress/pprice/dm-4692/meas_base setup meas_deblender 2016_01.0+6 b1901 setup minuit2 5.28.00.lsst2 b1901 b1902 setup ndarray 10.1+58 b1901 b1902 setup numpy 0.0.1+5 b1901 b1902 setup obs_subaru LOCAL:/tigress/pprice/dm-4692/obs_subaru setup obs_test tickets.DM-4692-g1533aee20f+1 b1901 setup pex_config 2016_01.0 b1901 b1902 setup pex_exceptions 2016_01.0 b1901 b1902 setup pex_logging 2016_01.0 b1901 b1902 setup pex_policy 2016_01.0 b1901 b1902 setup pipe_base 2016_01.0+6 b1901 setup pipe_tasks LOCAL:/tigress/pprice/dm-4692/pipe_tasks setup psfex 2016_01.0 b1901 b1902 setup pyfits 3.4.0 b1901 b1902 setup python 0.0.3 b1901 b1902 setup python_d2to1 0.2.12 b1901 b1902 setup pyyaml 3.11.lsst1 b1901 b1902 setup scons 2.3.5 b1901 b1902 setup sconsUtils 2016_01.0 b1901 b1902 setup skymap 2016_01.0+6 b1901 setup skypix 10.0+347 b1901 setup stsci_distutils 0.3.7-1-gb22a065 b1901 b1902 setup swig 3.0.2.lsst1 b1901 b1902 setup utils 2016_01.0 b1901 b1902 setup wcslib 5.13.lsst1 b1901 b1902 setup xpa 2.1.15.lsst3 b1901 b1902 setup pprice@tiger-sumire:/tigress/pprice/dm-4692 $ processCcd.py /tigress/HSC/HSC --rerun price/dm-4692 --rerun price/dm-4692 --id visit=1248 ccd=100 --clobber-config /tigress/pprice/lsstsw/miniconda/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment. warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.') : Loading config overrride file '/tigress/pprice/dm-4692/obs_subaru/config/processCcd.py' WARNING: Unable to use psfex: No module named extensions.psfex.psfexPsfDeterminer Cannot import lsst.meas.extensions.photometryKron: disabling Kron measurements Cannot enable shapeHSM ( File ""src/Utils.cc"", line 42, in std::string lsst::utils::getPackageDir(const std::string&) Package meas_extensions_shapeHSM not found {0} lsst::pex::exceptions::NotFoundError: 'Package meas_extensions_shapeHSM not found' ): disabling HSM shape measurements : Loading config overrride file '/tigress/pprice/dm-4692/obs_subaru/config/hsc/processCcd.py' : input=/tigress/HSC/HSC : calib=None : output=/tigress/HSC/HSC/rerun/price/dm-4692 CameraMapper: Loading registry registry from /tigress/HSC/HSC/rerun/price/dm-4692/_parent/registry.sqlite3 CameraMapper: Loading calibRegistry registry from /tigress/HSC/HSC/CALIB/calibRegistry.sqlite3 processCcd: Processing {'taiObs': '2014-03-28', 'pointing': 817, 'visit': 1248, 'dateObs': '2014-03-28', 'filter': 'HSC-I', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 100, 'expTime': 270.0} processCcd.isr: Performing ISR on sensor {'taiObs': '2014-03-28', 'pointing': 817, 'visit': 1248, 'dateObs': '2014-03-28', 'filter': 'HSC-I', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 100, 'expTime': 270.0} processCcd.isr: Applying linearity corrections to Ccd 100 processCcd.isr.crosstalk: Applying crosstalk correction processCcd.isr: Set 0 BAD pixels to 3147.74 processCcd.isr: Flattened sky level: 3847.800781 +/- 2114.507723 processCcd.isr: Measuring sky levels in 8x16 grids: 3884.324645 processCcd.isr: Sky flatness in 8x16 grids - pp: 15293.248379 rms: 1173.423587 processCcd.isr: Setting rough magnitude zero point: 34.678409 processCcd.charImage: Processing {'taiObs': '2014-03-28', 'pointing': 817, 'visit': 1248, 'dateObs': '2014-03-28', 'filter': 'HSC-I', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 100, 'expTime': 270.0} processCcd.charImage.repair: Identified 6044 cosmic rays. processCcd.charImage.detectAndMeasure.detection: Detected 127 positive sources to 5 sigma. processCcd.charImage.detectAndMeasure.detection: Resubtracting the background after object detection processCcd.charImage.detectAndMeasure.measurement: Measuring 127 sources (127 parents, 0 children) processCcd.charImage.measurePsf: Measuring PSF /tigress/pprice/lsstsw/stack/Linux64/meas_algorithms/tickets.DM-4692-g3d073a93d7+1/python/lsst/meas/algorithms/objectSizeStarSelector.py:354: RuntimeWarning: invalid value encountered in less bad = numpy.logical_or(bad, width < self._widthMin) /tigress/pprice/lsstsw/stack/Linux64/meas_algorithms/tickets.DM-4692-g3d073a93d7+1/python/lsst/meas/algorithms/objectSizeStarSelector.py:355: RuntimeWarning: invalid value encountered in greater bad = numpy.logical_or(bad, width > self._widthMax) processCcd.charImage.measurePsf: PSF star selector found 6 candidates meas.algorithms.psfDeterminer WARNING: You only have 3 eigen images (you asked for 4): reducing number of eigen components meas.algorithms.psfDeterminer WARNING: You only have 1 eigen images (you asked for 3): reducing number of eigen components meas.algorithms.psfDeterminer WARNING: You only have 1 eigen images (you asked for 2): reducing number of eigen components python: /tigress/pprice/lsstsw/stack/Linux64/eigen/3.2.5/include/Eigen/src/Core/Redux.h:202: static Eigen::internal::redux_impl<Func, Derived, 3, 0>::Scalar Eigen::internal::redux_impl<Func, Derived, 3, 0>::run(const Derived&, const Func&) [with Func = Eigen::internal::scalar_max_op<double>; Derived = Eigen::CwiseUnaryOp<Eigen::internal::scalar_abs_op<double>, const Eigen::Matrix<double, -1, -1> >; Eigen::internal::redux_impl<Func, Derived, 3, 0>::Scalar = double]: Assertion `size && ""you are using an empty matrix""' failed. Aborted {code} Note: * This occurred while testing DM-4692. The LOCAL pipe_tasks and obs_subaru are on that ticket branch. The LOCAL meas_base is for the fix from DM-5050. * One root cause of the bad PSF modeling may be bad rotations in the application of the calibs ([~lauren] is looking into that; don't know if there's a ticket number), but this should never happen regardless.",1 DM-5079,"Operations planning ","Draft Operations planning w.b.s. Create additional activity Diagrams, Draft DPPD Operations processing talk.",15 DM-5080,"January Management ","develop design and project management methods in conjunction with L1 design. Deal with ARI labor component General management of staff",15 DM-5084,"PropagateVisitFlags doesn't work with other pipeline components","{{PropagateVisitFlags}}, which was recently ported over from HSC on DM-4878, doesn't work due to some inconsistencies with earlier packages/tasks: - The default fields to transfer have new names: ""calib_psfCandidate"" and ""calib_psfUsed"" - We're not currently transferring these fields from icSrc to src, so those fields aren't present in src anyway. I propose we just match against icSrc for now, since it has all of the fields we're concerned with. - It makes a call to {{afw.table.ExposureCatalog.subsetContaining(Point, Wcs, bool)}}, which apparently exists in C++ but not in Python; I'll look into seeing which HSC commits may have been missed in that port.",1 DM-5085,"Please add a package that includes obs_decam, obs_cfht and all validation_data datasets","It would be very helpful to have an lsstsw package that added all supported obs_* packages (certainly including obs_cfht and obs_decam, and I hope obs_subaru) and all validation_data_* packages. This could be something other than lsst_apps, but I'm not sure what to call it.",1 DM-5086,"Enable aperture correction on coadd processing","Aperture corrections are now coadded, so we can enable aperture corrections in measurements done on coadds.",1 DM-5088,"Add auto play,select which dialog, close button working, to expanded mode","Add the auto play to expanded mode. Add the choose which dialog to expanded mode. Make close button work. I am breaking this up the expanded mode ticketa because the task is getting so big and ticket DM-5019 involved reducer refactoring. Also the refactoring needs to get into the dev branch.",4 DM-5089,"Add task discovery on command line activator","I'll add a way to specify on the command line the path or the package to discover for CmdLineTask or SuperTasks",4 DM-5090,"Investigate alternative for networkx before RFC","I'll make sure I explored other alternatives before creating a RFC for adding networkx which by itself require other packages. This is needed for the pipe_flow_x work. I tried one stand-alone package before pygraphviz but then decided to migrate to networkx as it is more complete and allow other possible future features",4 DM-5091,"LDM documentation of butler basics & multiple repositories",NULL,2 DM-5092,"Security plan renewal","Renewal of the LSST security plan. Starts with DM.",3 DM-5094,"HSC backport: Set BAD mask for dead amps instead of SAT","This is a port of [HSC-1095|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1095] and a leftover commit from [HSC-1231|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1231]: [isr: don't perform overscan subtraction on bad amps|https://github.com/HyperSuprime-Cam/obs_subaru/commit/d6fe6cf5c4ecadebd5a344d163e1f1e60137c7e4] (noted in DM-3942).",3 DM-5095,"Redirect confluence based pages to new developer guide.","Delete and apply redirects to all migrated pages in old Confluence-based Developer Guide",1 DM-5096,"Make validateDrp a Task.","Make validateDrp a Task so 1. it can easily be run from the command line or programmatically. 2. it can import the standard command line arguments 3. it can be logged in the same way. This eventually should fit into DM-2050, and DM-3859.",2 DM-5097,"Update validate_drp to use TransformTask to store calibrated measurements","Currently validate_drp uses some manual crude addition of calibration information and constructs new schemas to store this information. This is essentially what TransformTask is meant for. Using this would simplify the code, make it less fragile, and ideally eventually integrate more transparently with future calibration improvements or redefinitions of how zeropoints are tracked.. 1. Learn how to use TransformTask. Note DM-4948 is the doc task for this. 2. Adapt the code. 3. Verify unchanged results on existing validation_data_decam and validation_data_cfht.",2 DM-5098,"Add tests to validate_drp to verify SRD calculations and utility function behavior","The current validate_drp is woefully lacking in tests. 1. The key SRD metrics definitely need to have test cases that verify the calculation of these important metrics. 2. Overall the utility functions would benefit from testing.",4 DM-5099,"Polish IN2P3 cluster upgrade to CentOS7","What remains: - problem with Docker 1.9.1+overlay+xfs => switch to Docker 1.10.1? Then switch back from devicemapper to overlay? - problem with qserv uid: go back to 1000, instead of 1008?",4 DM-5100,"Docs for ltd-keeper","Create a documentation project within ltd-keeper that documents the RESTful API while it is being developed. This will allow the [SQR-006|http://sqr-006.lsst.io] technote to have a place to link to for detailed information.",1 DM-5101,"Fix --id examples in processCcd.py and friends to correctly show ""ccd=1^2"".","The required '^' convention for lists of things, e.g. {{ccd}}, {{filter}}, {{visit}} and such is surprising. But, worse, the documentation is currently wrong in its examples and presents several {{ccd=1,2}}, {{patch=1,2}} examples. * Fix the {{--id}} examples in {{pipe_tasks}} and other uses of processCcd.py in obs_* packages to correctly match the required syntax. Here's the current list in {{pipe_tasks}}, but check other packages as well. {code} [serenity tasks] grep '[0-9],[0-9]' *.py | grep '""' assembleCoadd.py: help=""data ID, e.g. --id tract=12345 patch=1,2"", coaddBase.py: parser.add_id_argument(""--id"", ""deepCoadd"", help=""data ID, e.g. --id tract=12345 patch=1,2"", imageDifference.py: parser.add_id_argument(""--id"", ""calexp"", help=""data ID, e.g. --id visit=12345 ccd=1,2"") makeDiscreteSkyMap.py: boxI = afwGeom.Box2I(afwGeom.Point2I(0,0), afwGeom.Extent2I(md.get(""NAXIS1""), md.get(""NAXIS2""))) multiBand.py: parser.add_id_argument(""--id"", ""deepCoadd"", help=""data ID, e.g. --id tract=12345 patch=1,2 filter=r"", multiBand.py: help=""data ID, e.g. --id tract=12345 patch=1,2 filter=g^r^i"") multiBand.py: parser.add_id_argument(""--id"", ""deepCoadd"", help=""data ID, e.g. --id tract=12345 patch=1,2 filter=r"", processCoadd.py: parser.add_id_argument(""--id"", ""deepCoadd"", help=""data ID, e.g. --id tract=12345 patch=1,2"", {code}",1 DM-5102,"Rewrite integration test queries with spatial constraint returning empty results","Some queries in the integration test suite return empty results, here's how to catch them: {code:bash} # this should be done for alll tests cases egrep ""^0$"" ~/qserv-run/2016_02/tmp/qservTest_case02/outputs/mysql/* # empty results files have also to be tracked {code} There parameters should be fixed to query a region containing data (use select * on object).",5 DM-5103,"Add scans for DRx-1 to the model","Per RFC-134 we need to support scans for DRx-1. This story involves building this into the model, costing it, and changing the baseline.",6 DM-5104,"Add scans for DRP-produced Dia* tables to the model","Per RFC-133, we need to support scans on DiaObject table, possibly Dia*Source tables as well. This story involves adding it to the model, costing it and adding it to the baseline.",6 DM-5105,"new conda 'mkl' dependent packages break meas_base tests","Continuum release/rebuilt a number of packages last friday to depend on the the Intel MKL library. https://www.continuum.io/blog/developer-blog/anaconda-25-release-now-mkl-optimizations There are [new feature named] versions that continue to use openblas but the MKL versions appear to be installed by default. This causes at least multiple {{meas_base}} tests to fail.After extensive testing, I have confirmed that the meas_base tests do not fail with the equivalent 'nomkl' package. In addition, mkl is closed source software that requires you to accept and download a license file or it is time-bombed to stop working after a trial period. {code:java} docker-centos-7: [ 36/36 ] meas_base 2015_10.0-9-g6daf04b+7 ... docker-centos-7: docker-centos-7: ***** error: from /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/build.log: docker-centos-7: tests/sincPhotSums.py docker-centos-7: docker-centos-7: tests/measureSources.py docker-centos-7: docker-centos-7: tests/testApertureFlux.py docker-centos-7: docker-centos-7: tests/testJacobian.py docker-centos-7: docker-centos-7: tests/testScaledApertureFlux.py docker-centos-7: docker-centos-7: The following tests failed: docker-centos-7: /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/meas_base-2015_10.0-9-g6daf04b+7/tests/.tests/sincPhotSums.py.failed docker-centos-7: /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/meas_base-2015_10.0-9-g6daf04b+7/tests/.tests/measureSources.py.failed docker-centos-7: /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/meas_base-2015_10.0-9-g6daf04b+7/tests/.tests/testApertureFlux.py.failed docker-centos-7: /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/meas_base-2015_10.0-9-g6daf04b+7/tests/.tests/testJacobian.py.failed docker-centos-7: /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/meas_base-2015_10.0-9-g6daf04b+7/tests/.tests/testScaledApertureFlux.py.failed docker-centos-7: 5 tests failed {code} (the exact cause of the test failures was not investigated as this should not have happened) This change has also broken the ability to import an existing conda env from 2016-02-05 or earlier that uses scipy due to some sort of package version resolution problem. Explicit declaring it as the scipy package without mkl fixes the resolution problem. There is a new 'nomkl' package, when installed, any subsequent package installations will default to versions without mkl. However, this does not fix any already installed packages. I am traumatized by the lack of reproducible build envs even within a few days of each other. After discussion in the Tucson office, I'm going to pin the lsstsw and newinstall.sh conda package versions with a commitment from square to update them on a monthly basis. I already have a test version of lsstsw/bin/deploy that defaults to a bundled package but with a option flag to use bleeding edge. ",4 DM-5106,"newinstall.sh fails with ""eups: command not found""","[~jgates] has reported the following output when running {{newinstall.sh}} on el6. {code:java} Installing EUPS (v2.0.1)... done. setup: No module named utils Installing Miniconda2 Python Distribution ... newinstall.sh: line 277: eups: command not found {code} Clearly, a command failure which should have been fatal was ignored. ",2 DM-5107,"Fix effective coordinates for defects in obs_subaru","The defects as defined in {{obs_subaru}} (in the {{hsc/defects/20NN-NN-NN/defects.dat}} files) are defined in a coordinate system where pixel (0, 0) is the lower left pixel. However, the LSST stack does not use this interpretation, preferring to maintain the coordinate system tied to the electronics. As such, the defect positions are being misinterpreted for the rotated CCDs in HSC (see [HSC CCD layout|http://www.naoj.org/Observing/Instruments/HSC/CCDPosition_20150804.png]). This needs to be remedied.",2 DM-5109,"Offset in gaussian-psf in ci_hsc","I'm seeing what looks like an aperture correction problem in psf-gaussian on {{ci_hsc}} coadds. This gets in the way of our ability to do star/galaxy classification, and suggests potentially more serious problems elsewhere. ",2 DM-5110,"S17 Data Access and Database Documentation","Update the documentation for Data Access and Database",65 DM-5111,"S17 Data Access and Database Documentation","Update the documentation for Data Access and Database",65 DM-5112,"FY17 Data Access Model Refresh","A refresh of the storage / IO model (LDM-141). The work involves understanding cost impact, and discussing the impact on science.",26 DM-5113,"S18 Data Access and Database Documentation","Update the documentation for Data Access and Database.",65 DM-5114,"S18 Data Access and Database Documentation","Update the documentation for Data Access and Database",65 DM-5115,"FY20 Data Access and Database Documentation","Update the documentation for Data Access and Database.",100 DM-5116,"FY18 Data Access Model Refresh","A refresh of the storage / IO model (LDM-141). The work involves understanding cost impact, and discussing the impact on science.",26 DM-5117,"FY19 Data Access Model Refresh","A refresh of the storage / IO model (LDM-141). The work involves understanding cost impact, and discussing the impact on science.",26 DM-5118,"FY18 Data Access Model Refresh","A refresh of the storage / IO model (LDM-141). The work involves understanding cost impact, and discussing the impact on science.",26 DM-5119,"FY19 Data Access and Database Documentation","Update the documentation for Data Access and Database.",100 DM-5120,"Add intelligence to `validate_drp` so it does ""A Reasonable Thing"" on an unknown output repo","validate_drp current takes as input both a repository and a configuration file. The configuration file contains information to construct the list of dataIds to analyze. However, these dataIds could be extracted from the repo itself, in cases where the desired is to analyze the entire repo. 1. Add a function that loads the set of dataIds from the repo. (/) 2. Select reasonable defaults for the additional parameters specified in the config file. (/) 3. Design how to handle multiple filters. (/)",5 DM-5121,"Add multiple-filter capabilities to `validate_drp`","Design and refactor `validate_drp` to produce results for multiple filters. 1. Decide on the syntax for the YAML configuration file that denotes the multiple filters. E.g., which visit goes with what filter? (/) 2. Organize the running of multiple filters in `validate.run` to sequentially generate statistics and plots for each filter. (/) 3. Add a filter designation to the default output prefix. (/) Note: matching objects *across* filters is out-of-scope for this ticket.",1 DM-5122,"LOAD DATA LOCAL does not work with mariadb","After we un-messed mariadb-mysqlclient we see errors now when trying to run integration tests: {noformat} File ""/usr/local/home/salnikov/dm-yyy/lib/python/lsst/qserv/wmgr/client.py"", line 683, in _request raise ServerError(exc.response.status_code, exc.response.text) ServerError: Server returned error: 500 (body: ""{""exception"": ""OperationalError"", ""message"": ""(_mysql_exceptions.OperationalError) (1148, 'The used command is not allowed with this MariaDB version') [SQL: 'LOAD DATA LOCAL INFILE %(file)s INTO TABLE qservTest_case01_mysql.LeapSeconds FIELDS TERMINATED BY %(delimiter)s ENCLOSED BY %(enclose)s ESCAPED BY %(escape)s LINES TERMINATED BY %(terminate)s'] [parameters: {'terminate': u'\\n', 'delimiter': u'\\t', 'enclose': u'', 'file': '/home/salnikov/qserv-run/2016_02/tmp/tmpWeAj6u/tabledata.dat', 'escape': u'\\\\'}]""}"") 2016-02-10 14:17:40,836 - lsst.qserv.admin.commons - CRITICAL - Error code returned by command : qserv-data-loader.py -v --config=/usr/local/home/salnikov/testdata-repo/datasets/case01/data/common.cfg --host=127.0.0.1 --port=5012 --secret=/home/salnikov/qserv-run/2016_02/etc/wmgr.secret --delete-tables --chunks-dir=/home/salnikov/qserv-run/2016_02/tmp/qserv_data_loader/LeapSeconds --no-css --skip-partition --one-table qservTest_case01_mysql LeapSeconds /usr/local/home/salnikov/testdata-repo/datasets/case01/data/LeapSeconds.schema /usr/local/home/salnikov/testdata-repo/datasets/case01/data/LeapSeconds.tsv.gz {noformat} It looks like mariadb client by default disables LOCAL option for data loading and it needs to be explicitly enabled. ",1 DM-5124,"Adapt all HSC calibration data to LSST camera geometry","In the [HSC CCD layout|http://www.naoj.org/Observing/Instruments/HSC/CCDPosition_20150804.png], approximately half of the HSC CCDs are rotated 180 deg with respect to the others, two others have 90 deg rotations and another two have 270 deg rotations. The HSC camera geometry defined a coordinate system where pixel (0, 0) is always the lower-left corner. However, the new camera geometry in the LSST stack does not use this interpretation, preferring to maintain the coordinate system tied to the electronics. As such, accommodations have had to be made for the rotated CCDs on {{obs_subaru}}. See DM-4998 and DM-5107 in particular for details. The need for these accommodations, and the accommodations themselves, should be removed. This entails a re-ingestion of the HSC calibration data files (BIAS, DARK, FLAT, etc.) as well as a redefinition of the defects files in {{obs_subaru}}.",4 DM-5125,"qserv fails when it mixes mariadb and mariadbclient directories","When I tried to run qserv-configure after installing qserv 2016_01-7-gbd0349f I got this error: {noformat} 2016-02-10 16:03:16,915 - lsst.qserv.admin.commons - CRITICAL - Error code returned by command : /home/salnikov/qserv-run/2016_02/tmp/configure/mysql.sh {noformat} Running script configure/mysql.sh: {noformat} $ sh -x /home/salnikov/qserv-run/2016_02/tmp/configure/mysql.sh + echo '-- Installing mysql database files.' -- Installing mysql database files. + /u2/salnikov/STACK/Linux64/mariadbclient/10.1.11/scripts/mysql_install_db --basedir=/u2/salnikov/STACK/Linux64/mariadbclient/10.1.11 --defaults-file=/home/salnikov/qserv-run/2016_02/etc/my.cnf --user=salnikov + echo 'ERROR : mysql_install_db failed, exiting' ERROR : mysql_install_db failed, exiting + exit 1 {noformat} and {noformat} $ /u2/salnikov/STACK/Linux64/mariadbclient/10.1.11/scripts/mysql_install_db --basedir=/u2/salnikov/STACK/Linux64/mariadbclient/10.1.11 --defaults-file=/home/salnikov/qserv-run/2016_02/etc/my.cnf --user=salnikov FATAL ERROR: Could not find mysqld The following directories were searched: /u2/salnikov/STACK/Linux64/mariadbclient/10.1.11/libexec /u2/salnikov/STACK/Linux64/mariadbclient/10.1.11/sbin /u2/salnikov/STACK/Linux64/mariadbclient/10.1.11/bin {noformat} So it looks for mysqld in mariadbclient, the same directory as mysql_install_db script, mysql_install_db should be actually running from mariadb. ",1 DM-5127,"FY18 Centralize access to database servers","We will have multiple services: L1 live database, multiple DR databases, calibration databases, EFD etc. It'd be nice if users would not have to know which server / which port / which dialect (plain mysql or qserv etc) to use. Instead, it'd be good to have a single entry point that redirects to the right place.",60 DM-5128,"Cost adding the support for Object / DiaObject joins in Qserv","Per RFC-133, we should support Object / DiaObject joins. That requires changes to query analyzer (and possibly elsewhere), currently we only support self-joins on objectId for director table. We'd need to either make DiaObject a director table and allow director-director joins, or allow director-child joins. This story involves costing how much effort it will be to implement it (and making a straw-man proposal how to implement it)",2 DM-5129,"Create InputField for generic use cases.","Create a composable, validating InputField so it can use outside of the form/submit use-case.",2 DM-5130,"B-F correction breaks non-HSC custom ISR, ci_hsc","The addition of brighter-fatter correction on DM-4837 breaks obs_cfht's custom ISR, since it slightly changes an internal ISR API by addding an argument that isn't expected by the obs_cfht version. It also breaks ci_hsc, since the B-F kernel file isn't included in the calibrations packaged there. ",1 DM-5131,"make the fits statistics call work with JSON",NULL,1 DM-5132,"obs_subaru install with eups distrib fails","Thus: {code} $ eups distrib install -t w_2016_06 obs_subaru ... [ 52/52 ] obs_subaru 5.0.0.1-60-ge4efae7+2 ... ***** error: from /Users/jds/Projects/Astronomy/LSST/stack/EupsBuildDir/DarwinX86/obs_subaru-5.0.0.1-60-ge4efae7+2/build.log: ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/hscRepository.py"", line 91, in setUp self.repoPath = createDataRepository(""lsst.obs.hsc.HscMapper"", rawPath) File ""tests/hscRepository.py"", line 63, in createDataRepository check_call([ingest_cmd, repoPath] + glob(os.path.join(inputPath, ""*.fits.gz""))) File ""/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py"", line 540, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['/Users/jds/Projects/Astronomy/LSST/stack/EupsBuildDir/DarwinX86/obs_subaru-5.0.0.1-60-ge4efae7+2/obs_subaru-5.0.0.1-60-ge4efae7+2/bin/hscIngestImages.py', '/var/folders/jp/lqz3n0m17nqft7bwtw3b8n380000gp/T/tmptUSKuf', '/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/testdata_subaru/master-gf9ba9abdbe/hsc/raw/HSCA90402512.fits.gz']' returned non-zero exit status 1 ---------------------------------------------------------------------- Ran 8 tests in 9.928s FAILED (errors=7) The following tests failed: /Users/jds/Projects/Astronomy/LSST/stack/EupsBuildDir/DarwinX86/obs_subaru-5.0.0.1-60-ge4efae7+2/obs_subaru-5.0.0.1-60-ge4efae7+2/tests/.tests/hscRepository.py.failed 1 tests failed scons: *** [checkTestStatus] Error 1 scons: building terminated because of errors. + exit -4 {code} Please fix it.",1 DM-5133,"Make meas_simastrom a stack package","Currently the simastrom code is sitting outside LSST which makes it not very visible and does not get built regularly making it sensitive to bitrot. Before we can really continue to gather requirements and develop the system, we need to bring it inside the fence. This should make the package located [here|https://github.com/lsst-france/meas_simastrom] buildable and usable in the LSST system with a minimum of external dependencies. By usable, I mean that it should be callable as a task. This does not need to solve the problem of persistence.",10 DM-5135,"Make ci_hsc buildable by Jenkins","1. Make sure {{ci_hsc}} is buildable by {{lsstsw}} / {{lsst_build}} (/) 2. Add {{ci_hsc}} to lsstsw/etc/repos.yaml so that one can request that Jenkins builds it. (/) 3. Verify that the test in {{ci_hsc}} fails on known broken tags and passes on known successful tags. (/) No dependencies will be added to {{lsst_sims}} or {{lsst_distib}}. This is meant to provide the ability to request that Jenkins do these builds and to fail if something has broken them. This will later be expanded to new packages {{ci_cfht}}, {{ci_decam}}, and {{ci_sim}}. The key goal is to make sure one hasn't broken obs_ packages in their butler interface or in their processCcd Additional Notes and Thoughts from HipChat Discussion [~ktl] Sounds good to me; we might have an ""lsst_ci"" top-level metapackage depending on all of them which is what Jenkins would run regularly. If the goal is to test obs_ packages, then my first instinct would be to put that in the obs_ package. Longer term goal to test the stack with different precursor datasets. If this is testing obs_ packages on a slower cadence than the built-in tests, it's OK for that to be a separate package. [~jbosch] Eventually, I think we need to run a CI dataset for each camera, then run some camera generic tests on each of those, then run some camera-specific tests on each of those.  So we don't want to go too far down a road in which all tests are camera-specific, but maybe we don't have a choice until we have some better unifying framework for them. I've certainly been putting some checks in {{ci_hsc}} that would be valid for all other cameras, if we had a CI package for them that went through to coadd processing.",2 DM-5136,"Increase key_buffer_size","I just looked at my qserv-run/etc/my.cnf and I don't see us setting key_buffer_size there. Looking at mysqld run as part of qserv I can see it is set to 128 MB. That is pretty low given we are planning to do lots of joins. Please add an entry in my.cnf that sets it to something higher with a comment that ""~20% of available RAM is recommended"".",1 DM-5137,"on-going support to Camera team in visualization (Feb. 2016) ","Attend the weekly meeting and answer questions as needed. Help with the Python and JS debug ",4 DM-5138,"document adding git-lfs repos to CI",NULL,1 DM-5139,"Update apr and apr_util","{{apr}} and {{apr-util}} are outdated and lagging behind the versions on RHEL6. They should be updated as agreed in RFC-76.",1 DM-5140,"Move luaxmlrpc to lsst-dm/legacy-","We no longer need luaxmlrpc because we run czar inside proxy. We should move it to lsst-dm/legacy-, and remove mentioning it in readme.",1 DM-5142,"DM Power Requirements Justification","The power requirements for the base site appeared to have increased greatly since LSE-239 or LDM-144 v140. Significant effort was spent digging through LDM-144 for precise rack counts, rack weights, rack power. Further time was spent on the analysis of why the power requirement is greater then expected. This involved analyzing swing space power requirements, max swing space needed, investigation into what LSE-239 refers to 'expansion' (turns out to be alert processing), attributing alert processing power requirements to the base (LDM-144 contributes to archive site but contingency is still in place for base site operations), comparing peak and steady state power needs. Also discussions around reinforcing the floors for greater rack weights. ",6 DM-5143,"Jason Feb Tasks","Weeks 1&2 - Interviews, Team mtgs, uptime institute tier discussions: 1.5 pts Weeks 3&4 - Team mtgs, ICI meetings, set/prioritize IT goals 4 pts",6 DM-5144,"Jason Feb Educational Activities","Learning DM stack deployment and layout, reading on redesign of butler 1.5",2 DM-5145,"Avoid merge table (i.e. result_m) creation on czar side","When launching a query which require an aggregation/merge, Qserv first creates a result_m table to collect chunk query results and then a result table. On the other hand, for a query which doesn't require a merge, only result table is created. If merge query was send to mysql-proxy right after query parsing (like it is currently done with ""order by"" clause only, this would be then generalized to all merge queries), creation of result_m table could be avoided. This would lead to simpler C++ code, and aggregation would be performed at the same time that returning result, which may lead to better performance. Queries wich requires or not aggregation step would be processed exactly the same way on the C++ side (store results of chunk queries), and mysql-proxy would release lock on result table when running aggregation/merge query (here, one can consider that simply concatening results of query would also be a kind of aggregation). Please not that removal of result_m table would also free some space on master, which is a bottleneck. [~jbecla], I propose you to plan this interesting feature for next sprint, feel free to postpone it. I think that intersection with ""Query coverage"" story might not be empty.",10 DM-5147,"Provide usable repos in {{validation_data_*}} packages.","Re-interpreted ticket: 1. Provide already-initialized repositories in the `validation_data_cfht`, `validation_data_decam`, and `validation_data_hsc` packages alongside the raw data. The goal is to allow both easy quick-start analyses as well as comparisons of output steps from processCcd.py and friends at each step of the processing. (/) 2. Add (Cfht,Decam,HSC).list files to provide for easy processing of the available dataIds in the example data. (/) 3. Update README files to explain available data. (/) [Original request:] In validation_drp when I run examples/runXTest.sh I find that any data I had saved in CFHT or DECam is lost, even if I have carefully renamed it. This is very dangerous and I lost a lot of work due to it. At a bare minimum please do NOT touch any directories not named ""input"" or ""output"". Lower priority requests that I hope you will consider: - Have the the input repo be entirely contained in the validation_data_X packages, ready to use ""as is"". That would simplify the use of those packages by other code. It would also simplify validate_drp, and it would just leave the output repo to generate (which already has a link back to the input repo). - Have runXTest.sh accept a single argument: the path to the output. (The path to the input is not necessary if you implement the first suggestion).",3 DM-5148,"IN2P3 cluster worker nodes failed to start due to Innodb error","Next error happens when starting mariadb on worker (with existing data from 35TB dataset, which were generated by mysql): {code:bash} 2016-02-13 22:02:36 139632684558144 [Note] InnoDB: Completed initialization of buffer pool 2016-02-13 22:02:36 139632684558144 [ERROR] InnoDB: auto-extending data file ./ibdata1 is of a different size 640 pages (rounded down to MB) than specified in the .cnf file: initial 768 pages, max 0 (relevant if non-zero) pages! 2016-02-13 22:02:36 139632684558144 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data! 2016-02-13 22:02:36 139632684558144 [ERROR] Plugin 'InnoDB' init function returned error. 2016-02-13 22:02:36 139632684558144 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 2016-02-13 22:02:36 139632684558144 [Note] Plugin 'FEEDBACK' is disabled. 2016-02-13 22:02:36 139632684558144 [ERROR] Unknown/unsupported storage engine: InnoDB 2016-02-13 22:02:36 139632684558144 [ERROR] Aborting {code}",4 DM-5150,"Create an easy place to add tests to ci_hsc","Create a single file where tests for validating source can be added. The tests will be duck typed to a class method and be registered to the corresponding validation class with a decorator.",1 DM-5151,"Code review, Feb 2016","DM3733,DM4825",2 DM-5152,"Meetings, Feb 2016","verification dataset meetings",1 DM-5153,"Process a tiny set of raw DECam Stripe 82 data","Process some DECam data to gain familiarity with process execution and learn to debug issues",8 DM-5154,"Continue learning middleware","Ramp up with the middleware status and development. Look into packages pipe_base, pex_config, pex_logging. ",16 DM-5155,"LOE, Feb 2016","Local LSST meetings, postdoc meetings, NCSA All hand meetings, RFDs, NCSA software meeting, astronomy events, workshops, travel to JTM, other local meetings. ",10 DM-5156,"Please document MemoryTestCase","{{lsst.utils.tests.MemoryTestCase}} is used extensively throughout our test suite, but it is lacking in documentation and it's not clear under what circumstances its use is required or encouraged. Please add appropriate documentation to the [Software Unit Test Policy |http://developer.lsst.io/en/latest/coding/unit_test_policy.html]. See also [this thread on clo|https://community.lsst.org/t/what-is-the-policy-for-using-lsst-utils-tests-memorytestcase].",1 DM-5160,"Record CCD, visit of input catalog in `validate_drp`","1. Record the CCD and `visit` of the individual source in the catalog so that it is available for later analysis. 3. Update `analyzeData` to use these newly available CCD and `visit` information in the catalog. ",1 DM-5161,"HSC backport: Support a full background model when detecting cosmic rays","This is a port of the following two standalone HSC commits: [Support a full background model when detecting cosmic rays|https://github.com/HyperSuprime-Cam/pipe_tasks/commit/3bae328e0fff4b2a02267e97cc1e53b5bbe431cb] {code} If there are strong gradients (e.g. M31's nucleus) we need to do more than treat the background as a constant. However, this requires making a copy of the data so the background-is-a-constant model is preserved as a special case {code} [Fixed cosmicRay() in RepairTask for the case background is subtracted.|https://github.com/HyperSuprime-Cam/pipe_tasks/commit/2cdb7c606270d84c7a05baf9949ff5724463fa6b] {code} When the background is subtracted with finer binsize, new exposure will be created and cosmic rays will be detected on that exposure. But the image of that exposure was not properly returned back. {code} ",1 DM-5162,"Audit the LSST and HSC codebases for differences","We've already merged a lot of code from HSC to LSST, and are optimistic that we've captured most of the big ticket items. However, we need to perform a thorough comparison of the codebases to check there's nothing we're missing. Please do that, and file tickets in the DM-3560 and DM-3568 epics to describe outstanding work.",4 DM-5163,"Modify System layout to support expanded views","Each of the visualizers needs to expand to full screen. We need to modify our current layout system so each and expand and collapse so that the old view is restored. The system needs to be flexible enough so an 'expanded version' of the component can be used.",4 DM-5164,"Tests in daf_persistence should skip properly","Some of the tests in {{daf_persistence}} have a couple of problems that cause difficulties with modern test frameworks: # unittest is not being used at all in some cases # Skipping is done with a print and a {{sys.exit}} They need to be modernized.",2 DM-5165,"Mouse Readout: part 1.5 - update flux server call to work in JS",NULL,1 DM-5166,"Analyze catalog-comparison CmdLineTasks","Analyze the QA CmdLineTask collection being generated by [~lauren] sufficiently well to determine the interface requirements needed to represent them as Supertasks. Does not include actually designing that interface.",6 DM-5167,"Standup Fastly infrastructure for LSST the Docs","LSST the Docs will use Fastly to serve docs out of an S3 bucket with well-formatted URLs thanks to routing at the Varnish layer. See https://www.hashicorp.com/blog/serving-static-sites-with-fastly.html for an overview of the desired setup and http://sqr-006.lsst.io for an overview of LSST the Docs. Specific outcomes are: * Create S3 bucket for LTD * Create Fastly account (may be a demo account pending negotiations with Fastly) * Basic configurations for Fastly account * Research pricing/configure a TLS certificate for *.lsst.io domains * Set up base VCL configuration on Fastly. ",3 DM-5169,"Fastly API interactions for LSST the Docs","Using Fastly’s API, have ltd-keeper setup new builds and editions: - Add {{Surrogate-Key}} to headers of objects uploaded to S3 (happens on ltd-mason side) - Configure Varnish to serve specific bucket directories as specific domains (DM-4951 has added Route 53 interactions to ltd-keeper) - Purge content when editions switch or content is deleted. DM-5167 is covering non-API driven work to configure fastly. See https://www.hashicorp.com/blog/serving-static-sites-with-fastly.html for a write-up on serving static site via fastly. See also http://sqr-006.lsst.io for an overview of LSST the Docs.",8 DM-5171,"New XY functions to be developed (F16)","There are several new functions requested by users",45 DM-5174,"Manipulating masks is confusing","A possible bug exists in afwImage.Exposure.getMaskedImage(). This function returns a copy of the Exposure's masked image, and not the actual maskedImage owned by the Exposure. This means that any changes made to the mask are done only on the copy, and are not reflected in the Exposure's maskImage. The intended behavior seems to be that a shallow copy be returned with pointers to all the original objects (such as the mask). This however does not seem to be the case, a deep copy is always made instead. Verify that the intended behavior is indeed happening. Steps to reproduce {code:python} coaddExposure = afwImage.ExposureF() coaddExposure.getMaskedImage().getMask().addMaskPlane('TEST') print(coaddExposure.getMaskedImage().getMask().getMaskPlaneDict().asdict()) m = coaddExposure.getMaskedImage().getMask() print(m.getMaskPlaneDict().asdict()) m.removeAndClearMaskPlane('TEST') print(m.getMaskPlaneDict().asdict()) print(coaddExposure.getMaskedImage().getMask().getMaskPlaneDict().asdict()) {code} A second concern, though not necessarily a bug, is that adding and removing mask planes is confusing due to inconsistent manipulation of global state. For example: {code} In [1]: import lsst.afw.image as afwImage # Create two separate Masks In [2]: mask1 = afwImage.MaskU() In [3]: mask2 = afwImage.MaskU() # Neither Mask contains a ""TEST"" plane In [4]: 'TEST' in mask1.getMaskPlaneDict() Out[4]: False In [5]: 'TEST' in mask2.getMaskPlaneDict() Out[5]: False # Adding a plane to one updates a shared list of planes, so it appears in the other In [6]: mask1.addMaskPlane('TEST') Out[6]: 9 In [7]: 'TEST' in mask1.getMaskPlaneDict() Out[7]: True In [8]: 'TEST' in mask2.getMaskPlaneDict() Out[8]: True # But deleting a plane from one affects only that particular Mask and not the global state In [9]: mask1.removeAndClearMaskPlane('TEST') In [10]: 'TEST' in mask1.getMaskPlaneDict() Out[10]: False In [11]: 'TEST' in mask2.getMaskPlaneDict() Out[11]: True {code}",4 DM-5175,"Add CSS information for shared scans to integration test data.","Some tables int the integration tests need to be flagged as needing to be locked in memory and given a scan rating.",1 DM-5178,"lsstsw deploy on OS X fails in miniconda install","Testing the fixes for the {{deploy}} script in DM-4359 it seems that the part of the script installing {{miniconda}} no longer works on OS X because the list of packages to be installed has been derived from a Linux system and not all the Linux packages have OS X equivalents. There needs to be a per-OS list of packages. The default OS X list seems to be: {code} # This file may be used to create an environment using: # $ conda create --name <env> --file <this file> # platform: osx-64 astropy=1.1.1=np110py27_0 conda=3.19.1=py27_0 conda-env=2.4.5=py27_0 cycler=0.9.0=py27_0 cython=0.23.4=py27_1 freetype=2.5.5=0 libpng=1.6.17=0 matplotlib=1.5.1=np110py27_0 nomkl=1.0=0 numpy=1.10.4=py27_nomkl_0 openssl=1.0.2f=0 pandas=0.17.1=np110py27_0 pip=8.0.2=py27_0 pycosat=0.6.1=py27_0 pycrypto=2.6.1=py27_0 pyparsing=2.0.3=py27_0 pyqt=4.11.4=py27_1 python=2.7.11=0 python-dateutil=2.4.2=py27_0 pytz=2015.7=py27_0 pyyaml=3.11=py27_1 qt=4.8.7=1 readline=6.2=2 requests=2.9.1=py27_0 scipy=0.17.0=np110py27_nomkl_0 setuptools=19.6.2=py27_0 sip=4.16.9=py27_0 six=1.10.0=py27_0 sqlalchemy=1.0.11=py27_0 sqlite=3.9.2=0 tk=8.5.18=0 wheel=0.29.0=py27_0 yaml=0.1.6=0 zlib=1.2.8=0 {code}",1 DM-5179,"miniconda2 eups package fails to install on OS X","The {{miniconda2}} eups package attempts to install the relevant conda packages by downloading a list from the {{lsstsw}} repository. This fails for the same reason that {{lsstsw}} fails in DM-5178 in that the list of packages is not OS-specific. This means that {{newinstall.sh}} does not work any more on OS X.",1 DM-5181,"update ""newinstall.sh"" nebula images & docker containers - w_2016_08",NULL,1 DM-5182,"Hook up help system","We need to help system like we have in GWT.",8 DM-5185,"Implement Lock plot button on toolbar","* Write a button on the toolbar that monitors the active plot view's group and shows the locked or unlocked icons * Add an action and reducer functions the will toggle the lock state of the group.",6 DM-5186,"Add Xrdssi plugin configuration file","Xrdssi plugin configuration file could be useful for sharedscan. to pass plugin configuration file path to xrootd http://xrootd.org/doc/dev42/xrd_config.htm#_Passing_Plug-In_Command (use -+xrdssi) to get this argument from C++ http://xrootd.org/doc/dev42/ssi_reference.htm#_Toc431242001 then an add-hoc config file parser needs to be used (not to be xrootd dependant), json/yaml could be used.",6 DM-5187,"Set Qserv master in env variable for Docker containers","This would allow use of pre-configured container on all clusters, indeed the only parameter which currently change in cluster install is master fqdn. See http://xrootd.org/doc/dev42/Syntax_config.htm and {code} if defined ?~EXPORTPATH set exportpath = $EXPORTPATH else set exportpath = /tmp fi all.export $exportpath {code}",3 DM-5188,"Add fftools API: Image Viewer plus foundational work",NULL,10 DM-5189,"Add fftools API: Table",NULL,6 DM-5190,"Add fftools API: XYPlots and Histgram",NULL,8 DM-5191,"Coverage, Coverage API, ImageMetaData API",NULL,8 DM-5192,"Add remote (python) API support ","The python interface needs to be ported. This involves the following: * Modify FireflyClient.py * Change all the API to work by dispatching remote actions. There is currently a dispatchRemoteAction method in FireflyClient.py * On the server side clean up file PushCommands.java, PushJob.java and ServerParams to remove the old api. * Move the fftools/python to firefly/python * clean up the test notebooks. There are currently several, some don't work and should be removed. Others should be clean test cases. * Make sure the python support for RangeValues is consistent with the Java and JavaScript. I suspect it is not. * Make sure events can be received into the python. ",16 DM-5193,"attend the bi-weekly meeting authentication and authorization discussion","attend the bi-weekly meeting authentication and authorization discussion. provide input and feedback to IAM. ",4 DM-5194,"Deploy ltd-keeper as a Docker Container","ltd-keeper should be deployed as a Docker container as a best practice for maintainable cloud microservices. This involves writing a Dockerfile committed to the ltd-keeper repo and demonstrating that the container can be stood up on Google Container Engine. I plan on use data-containers attached to the service’s container to maintain the sqlite DB. This ticket should document how to operate ltd-keeper and apply updates to both the ltd-keeper app *and* DB migrations.. This ticket also involves initial overhead in researching Docker/Kubernetes.",12 DM-5196,"swift API availability?","The downtime announcement email for {{Nebula unavailable Feb 9-10}} mentioned a ""roadmap"" for swift. I have checked and post maintenance, there is not a swift endpoint available in the catalog. Is there a time line for availability?",1 DM-5197,"Test and robustify shapelet PSF approximations","The CModel code ported from HSC only works as well as the ShapeletPsfApproximation algorithm that runs before it, but we've switched on the LSST side to a more flexible algorithm that isn't as nearly as battle-tested as what's been running on the HSC side, and there are some concerning indications from [~pgee]'s work that it can be catastrophically slow on some reasonable PSFs. On this issue, I'll run it on some real HSC data and try to improve it, even if that means reducing the flexibility back to what was on the HSC side in some ways.",8 DM-5198,"FITS Visualizer porting: Statistics - part 2 - drawing overlay & 3 color support","drawing overlay 3 Color Support",8 DM-5200,"instance I/O errors","The kernel dmesg for Instance {{bbfd7458-6dd6-4412-a8ba-8d417c3df56b}} has started reporting thousands of block I/O errors and these are starting to trickle up as a filesystem I/O errors. I suspect this is likely a hypervisor I/O issue. {code} [687301.556430] Buffer I/O error on device dm-3, logical block 3768490 [687301.556433] Buffer I/O error on device dm-3, logical block 3768491 [687301.556436] Buffer I/O error on device dm-3, logical block 3768492 {code} {code} $ openstack server show bbfd7458-6dd6-4412-a8ba-8d417c3df56b +--------------------------------------+-----------------------------------------------------------------------+ | Field | Value | +--------------------------------------+-----------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-STS:power_state | 1 | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2016-02-11T23:36:25.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | LSST-net=172.16.1.115, 141.142.209.121 | | config_drive | | | created | 2016-02-11T23:36:12Z | | flavor | m1.xlarge (5) | | hostId | f7fbf308022d02f52e1111c91cf578d852784d290d0e0ddb0d69635c | | id | bbfd7458-6dd6-4412-a8ba-8d417c3df56b | | image | centos-7-docker-20151116230205 (59a2a478-11ab-41c5-affc-29706d38d65a) | | key_name | vagrant-generated-comshorc | | name | el7-docker-jhoblitt | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | project_id | 8c1ba1e0b84d486fbe7a665c30030113 | | properties | | | security_groups | [{'name': 'default'}, {'name': 'remote SSH'}] | | status | ACTIVE | | updated | 2016-02-11T23:36:25Z | | user_id | 83bf259d1f0c4f458e03f9002f9b4008 | +--------------------------------------+-----------------------------------------------------------------------+ {code}",1 DM-5201,"sph-partition does not support BLOB fields","Next command fails with BLOB field in input file: {code:bash} dev@clrinfoport09:~/src/qserv_testdata/datasets/case01/data$ sph-partition --out.dir /home/dev/qserv-run/git/tmp/qserv_data_loader/Object --part.prefix chunk --config-file /home/dev/src/qserv_testdata/datasets/case01/data/common.cfg --config-file /home/dev/src/qserv_testdata/datasets/case01/data/Object.cfg --in Object.tsv CSV record contains an embedded line terminator, a trailing escape character, or a quoted field without a trailing quote character. {code} Note that the command works in input file is reduced to its first line. Note that mysql import works fine: {code:sql} MariaDB [qservTest_case01_mysql]> LOAD DATA INFILE ""/tmp/Object.tsv"" INTO TABLE Object FIELDS TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '""' LINES TERMINATED BY '\n'; {code}",15 DM-5202,"Remove LOGF macros from log package","We have removed all uses of LOGF macros from qserv and as far as I know no other clients use those macros. It's time to clean up log package itself from those macros.",1 DM-5203,"Add support for 3 Color","Most of this is done. I just need to plot a few 3 color images and work out the bugs.",4 DM-5204,"Remove remaining LOGF macros from qserv","There are still few cases of LOGF macros in qserv, have to replace them all.",1 DM-5205,"replace Associations::CollectRefStars with LoadReferenceObjectsTask","AstroUtils.cc has code for loading USNO catalog which is used by Associations:CollectRefStars to build a reference list. We should replace this with LoadReferenceObjectsTask from meas_algorithms to both make it more generic, and to remove problematic endian handling in AstroUtils.",10 DM-5206,"Please do not write garbage to the FITS EQUINOX","The equinox is not relevant when dealing with ICRS coordinates. When {{afw}} manipulates {{Wcs}} objects, it simply doesn't bother initializing the {{equinox}} field of its {{_wcsInfo}} struct when dealing with an ICRS system. When {{afw}} persists the {{Wcs}} to FITS, it blindly writes whatever happens to be in that uninitialized field to the FITS header. Thus, we end up with something like: {code} EQUINOX = 9.87654321E+107 / Equinox of coordinates {code} This should be no problem, since, per the [FITS standard|http://fits.gsfc.nasa.gov/standard30/fits_standard30aa.pdf] (page 30), the {{EQUINOX}} is ""not applicable"" if they {{RADESYS}} is {{ICRS}}. The reader should thus ignore this value. However, [SAOimage DS9|http://ds9.si.edu] version 7.4.1 (the latest release at time of writing) does _not_ ignore the {{EQUINOX}}. Rather, it refuses to handle the WCS for the image. Note that version 7.3 of DS9 does not seem to have the same issue. While this does seem to be a bug in DS9, it's easy enough to work around by simply not writing {{EQUINOX}}.",1 DM-5208,"Evaluate MariaDB GSSAPI Authentication Plugin","As a follow-on to DM-4315, deploy the new [Maria DB GSSAPI Authentication Plugin|https://mariadb.com/kb/en/mariadb/gssapi-authentication-plugin/] in the IAM testbed for Kerberos ticket-based authentication, to provide single sign-on access.",2 DM-5209,"Improve worker configuration files.","Configuration on the worker is done by setting environment variables in a script, which is lacking in flexibility, but there is a question of if it is worth changing to some form of text configuration file. The code that reads the configuration could be improved in either case.",10 DM-5218,"Run Qserv multinodes integration tests inside Travis","This aims at preparing integration of this procedure inside Jenkins CI",4 DM-5219,"Add configured requirements parameters. Pass/Fail test.","1. Add pass/fail routine to report success/fail against metrics. Do this for * SRD (/) * Configured metrics (/) 2. Add pass/fail reporting to running of `validate.drp.run` (/)",4 DM-5222,"Add a ci_hsc daily build","Please add a daily build of `ci_hsc` to the Jenkins system. This does not need to explicitly build `lsst_distrib` or `lsst_sims`. The only product to list is `ci_hsc`. In the slightly near future, I anticipate that this build will be replaced by a daily build of a metapackage `lsst_ci`.",5 DM-5233,"implement cycle change in DLP","Summer --> Fall, Winter --> Sprint, add X16",1 DM-5246,"X16 Object Characterization Bucket","Catch all epic for essential bugs and improvements in object characterization emerging during X16.",20 DM-5247,"Segfault in shapeHSM centroid extractor","[~boutigny] reports a segfault in {{meas_extenstions_shapeHSM}}. He provides the following backtrace: {code} Program received signal SIGSEGV, Segmentation fault. 0x00007fffe7043156 in lsst::afw::table::BaseRecord::getElement<lsst::afw::table::Flag> (this=0x21c8d60, key=...) at include/lsst/afw/table/BaseRecord.h:61 61 typename Field<T>::Element * getElement(Key<T> const & key) { (gdb) bt #0 0x00007fffe7043156 in lsst::afw::table::BaseRecord::getElement<lsst::afw::table::Flag> (this=0x21c8d60, key=...) at include/lsst/afw/table/BaseRecord.h:61 #1 0x00007fffdc8775f2 in set<lsst::afw::table::Flag, bool> (value=<synthetic pointer>, key=..., this=0x21c8d60) at /home/boutigny/CFHT/lsstsw/stack/Linux64/afw/11.0-8-g38426eb/include/lsst/afw/table/BaseRecord.h:137 #2 setValue (value=true, i=0, record=..., this=0x1da2500) at include/lsst/meas/base/FlagHandler.h:73 #3 lsst::meas::base::SafeCentroidExtractor::operator() (this=<optimized out>, record=..., flags=...) at src/InputUtilities.cc:134 #4 0x00007fffd03655c6 in lsst::meas::extensions::shapeHSM::HsmPsfMomentsAlgorithm::measure (this=0x1da2410, source=..., exposure=...) at src/HsmMoments.cc:115 #5 0x00007fffd06708d5 in _wrap_HsmPsfMomentsAlgorithm_measure (args=0x7fffccc67b90) at python/lsst/meas/extensions/shapeHSM/hsmLib_wrap.cc:14337 #6 0x00007ffff7aee37f in ext_do_call (nk=-859407472, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffff7d18, func=0x7fffd0c21878) at Python/ceval.c:4345 #7 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2720 #8 0x00007ffff7aefdbe in PyEval_EvalCodeEx (co=0x7fffd0a9ceb0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=3, kws=0x7fffccd43b08, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3267 {code} See the discussion at DM-4780.",2 DM-5248,"Implement background gradient fit in pre-sub. images for dipole fit","Add a linear background gradient fit to the integrated pre-subtraction and dipole fitter (for testing). This will eventually be implemented in the measurement plugin.",2 DM-5249,"Implement a way to pass more than one exposure to a SingleFrameMeasurement (DipoleMeasurementTask)",NULL,10 DM-5250,"Add background gradient fit to new dipole measurement task",NULL,2 DM-5251,"lsstsw breakage with spaces in paths","There are still some issues relating to using {{lsstsw}} to build the stack when spaces are in the path to the {{$LSSTSW}} location. This is a fine thing to sort out on Rodeo Day...",1 DM-5252,"Base ""bright star"" cut on S/N instead of magnitudes","The astrometry histogram generated by validateDrp.py conflates astrometric and photometric calibration because it uses magnitude for brightness, and this relies on the accuracy of the photometric calibration. [~ctslater] suggests (and I agree) that brightness should be based on signal to noise ratio, thus making the astrometry histogram independent of photometric calibration. ",2 DM-5262,"X16 Butler","Work on improving Butler: * Refactor butler multiple repository support based on user feedback. * Formalize butler configuration mechanism and define configuration persistence. * RFC and implement support for repo selection based on version. * Design and RFC mechanism for Butler to define output dataset type. * R&D & stub implementation of Butler storage factorization (python type + file type + storage location) * implement spatial lookups in butler * minor bug fixing",49 DM-5264,"Modernize python in lsst_build","The python in {{lsst_build}} uses old-style print and exception handling. These should be updated to the current standard.",1 DM-5265,"Turn on bias-jump fix for all CCDs ","The overscan fix to handle bias jump in an amplifier done in DM-4366 introduced a new config parameter {{overscanBiasJumpBKP}}, and the fix is applied for CCDs on the backplanes specified in {{overscanBiasJumpBKP}}. Previously, the default is to only fix CCDs on backplanes next to the focus chips. But [~mfisherlevine] also see the bias jump features in other CCDs. It would make more sense to turn it on for all CCDs by default. ",1 DM-5267,"Prepare narrative description of Level 3 operations from the perspective of the SUIT","Also known as the ""Level 3 ConOps"" needed by NCSA.",10 DM-5270,"Provide comparison routines for comparing two repos of the same data","Adapt the HSC capabilities from DM-4730 as represented on pipe_tasks u/laurenm/DM-4730 (as prepared by [~lauren] and [~price]) into generally available {{pipe_tasks}} routines for comparison of two different repositories of the same data. The intended use case is comparing two different algorithms or configurations on the same data and providing individual source-measurement to source-measurement comparisons for debugging new algorithms and understanding the behavior.",3 DM-5271,"Audit SuprimeCam policy and update to current standards","{{obs_subaru}}'s {{policy/SuprimecamMapper.paf}} contains a number of entries that look wrong (e.g. {{deep_forcedPhotCoadd_metadata}} should be {{deepCoadd_forced_config}}) or do not apply to LSST (e.g. {{stack_config}}) and doesn't contain a number of entries that might be expected (e.g. {{transformed_src}}). Please ensure that this file is updated to comply with current expectations.",1 DM-5273,"rename meas_simastrom to jointcal and flatten namespace","Moving meas_simastrom from lsst_france/ to lsst/ also resulted in a name change per RFC-123, and a namespace flattening: it's not derived from meas; it's a task. This is the necessary first step in getting it integrated into the stack.",1 DM-5274,"Filtering from XY Plot table view (JS)","Allow to filter in a selected area from XY Plot.",4 DM-5275,"make floating point exception handling cross-platform (or remove it)","jointcal currently has a couple of trapfpe() functions that wrap feenableexcept, which doesn't exist on OSX. Were these an important part of error handling in meas_simastrom, or can I just remove them?",2 DM-5276,"plan to upgrade the third party packages","The following packages need to be reviewed and maybe upgraded. ""babel"" : ""5.8.34"", 6.5.2 (M) ""history"" : ""1.17.0"", 2.0.0 (M) ""icepick"" : ""0.2.0"", 1.1.0 (M) ""react-highcharts"": ""5.0.6"", 7.0.0 (M) ""react-redux"": ""3.1.2"", 4.4.0 (M) ""react-split-pane"": ""0.1.22"", 2.0.1 (M) ""redux-thunk"": ""0.1.0"", 1.0.3 (M) ""redux-logger"": ""1.0.9"", 2.6.1 (M) ""validator"" : ""4.5.0"", 5.1.0 (M) ""chai"": ""^2.3.0"", 3.5.0 (M) ""esprima-fb"": ""^14001.1.0-dev-harmony-fb"", 15001.1001.0-dev-harmony-fb (M) ""babel-eslint"" : ""^4.1.3"", 5.0.0 (M) ""babel-loader"" : ""^5.3.2"", 6.2.4 (M) ""babel-plugin-react-transform"": ""^1.1.0"", 2.0.0 (M) ""babel-runtime"" : ""^5.8.20"", 6.6.0 (M) ""eslint"" : ""^1.10.3"", 2.2.0 (M) ""eslint-config-airbnb"": ""0.1.0"", 6.0.2 (M) works with eslint 2.2.0 ""eslint-plugin-react"": ""^3.5.1"", 4.1.0 (M) works with eslint 2.2.0 ""extract-text-webpack-plugin"": ""^0.8.0"", 1.0.1 (M) ""html-webpack-plugin"": ""^1.6.1"", 2.9.0 (M) ""karma-sinon-chai"": ""^0.3.0"", 1.2.0 (M) ""redux-devtools"" : ""^2.1.2"", 3.3.1 (M) ""webpack"": ""^1.8.2"" 1.12.14, 2.1.0 beta4 (M) ",2 DM-5277,"replace buildbot with jenkins job(s)","Removing buildbot and replacing it with jenkins would provide a number of benefits * one less dashboard for developers to know about / interact with * one less system for SQRE to maintain * lessening the cost of refactoring the CI drivers scripts as synchronized updates to two CI system configurations would no longer be necessary It should also be easy to go one step further and try to eliminate the need for developers to manually log into the {{lsstsw}} account on {{lsst-dev}} to publish eups distrib packages. ",3 DM-5278,"Attend JTM","Joint Technical Meeting 2/22-24, Santa Cruz",6 DM-5279,"arrays not properly transmitted","Sending a property set with an array as one of the entries only passes the last element of the array.",1 DM-5280,"Port HSC afw changesets to LSST","We identified in DM-5162 several changesets that still need to be ported from HSC to LSST: * 2c12255372bde846ba0429b5b542960e57d169f0, 0aec617e0ea604cde85105de3dade279a4fe10df: Footprint::overlapsMask * 76f3706f6688b23d5b0c71e66af3e94095a9f821: copyWithinFootprint: respect image size * f49676d7f1348f9de8ca21ee633e0c25473251ae: Implemented Linear and ZScale transformations, HSC-1206, 08a7740ed756ee7b2c845b4ce6aed8d9a0f50d04 * c369a8ad53962aba950f7710210be4b23f45a523: utils: make Mosaic.nImage a property * Maybe 0a2647a4f57addc3b3adb347da995fa0d36b43cb: Add display.utils function to plot the bboxes of inputs to a coadd in ds9 * 386a4b71d974d9e5672fe8690d0db3e56c9fb40f: Box2?.getCorners * f3d3029c561b957069cbf280b62ea8e37447c068: Calib: add operator*= for scaling zeropoint * 6c1845474f28f528a95190eeb88f095b11999078: Check in #3092 (iterable coord) directly on master * f3d3029c561b957069cbf280b62ea8e37447c068: Calib: add operator*= for scaling zeropoint * 5d1934cc9fe7d8c43aa8f9318a1ac9a3ce85e94e, 0d1ab12db604d5e42a5d72f028411a64294283ce: Footprint merging * b8578746d69920bc1e1089cca4b4acb230f0e8d5: Interpolate: add support for ndarray * 3de3339aa075f869d73a5bc66fc65dbee8ae3d16: Fix unit test fallout from Interpolate std::vector change * 08a7740ed756ee7b2c845b4ce6aed8d9a0f50d04, f73544e15abd2760bf84794798cf4b84e97e938d: Added xSize, ySize, and rescaleFactor arguments to {make,write}RGB; HSC-1207 * 88d838b74898d9572bbc8c46121da029958c1c72: rgb: fix makeRGB so it can replace saturated pixels and produce an image * 254d7248ea20d98de481d968f6503d1610b16ae7: Remove tests of writing rgb images to jpeg and tiff * 3252a40ad55222b882acf14d2f7cf0f3fe80f585: Added MatchControl and implemented it for matchXY and MatchRaDec * 81c6063a32883b748f3770b7124d74ced7b480f5: Implement and test includeMismatches option in MatchControl object * fd4c0baec617155fac0816607a5acba88e8970f5: Add support for renaming without replacing the full field in SchemaMapper * 79337bb6d1ee3a0b73bcd2b2d0ca506a44d3fa56: Handle empty Footprints when normalising ",4 DM-5281,"Port HSC skymap, shapelet changesets to LSST","We identified in DM-5162 some changesets that still need to be ported from HSC to LSST: * skymap: ** f83f71718eac5307d575d3113ee3757a63a16de2: Set vertex list in ExplicitTractInfo. * shapelet: ** bb928df3fc2fafe5183e0d075da19994f0af4fc7: Let the value to normalize to be specified in [Multi]ShapeletFunction ",1 DM-5282,"Port HSC meas_algorithms changesets to LSST","We identified in DM-5162 several changesets that still need to be ported from HSC to LSST: * 1293a31c19c238ba2c2acd8f67ec1be742764b66: BinnedWcs * 9f392b134502f6e4fbbd8759806b15f89a267e5a: detection: additional debugging plots for background * ad74fe8595ec523d6269e36ec2db051534bf3e9a: Add initial.flags.pixel.interpolated.any to ObjectSizeStarSelectorConfig.badFlags * 69f5db0eba69225cff917fa4c96a94dc8b765aa0, 4a0d59e191fc40d3091b56b20cf27ede4e0c23ab: Check for bad PSF measurements (HSC-1153) * a54b1ac52678025d3317e8a379c2849d3eb567ba: pcaPsfDeterminer: catch case of no good PSF candidates in debugging * c4fcab3251e6f41da2248d63fdf28c0bf80e30f8: Indent seems to be wrong for debug display * 2a889c17d47c879dbb4345bafba6aed9869b5984, f3e42cc03ab8a4f1b28d9e0852619cbdbf3b7018: Make IdSpanCompar more deterministic * f99eb46f484609673b45290eaaba47688d7b4a24: CR code has to take care of 'NO_DATA' mask * 6f6b786bce8ca34bf4c67f75f965130dea027147: Handle small numbers of psfCandidates (HSC-1176) * d744e6514feaf67b87068ac502bca677306f9fc2: tests: add test for MeasureApCorrTask * 65f617089038fe19179fca4f959bf23ea061a6b8, 1b7e3cc48ed347b0afa31e81c821b38f87d18d64: Test case for measurement of negative objects There are also a couple of issues that were identified in the DM-5162 review: * Delete tests/config/MeasureSources.py --- mere configuration, old-style measurement * testPsfDetermination has method 'xtestRejectBlends'",4 DM-5283,"Port HSC daf_butlerUtils changesets to LSST","We identified in DM-5162 several changesets that still need to be ported from HSC to LSST: * daee24edba01b01a0412df7f9b4cf70be5b10860: CameraMapper: allow a default filter name to be provided * e3fee95d6a1850dd2309d3ebe4e3ef3ffe38eef0: CameraMapper: normalize path names, and remove leading double slash * 476b6ddccd9d0cceb2b89ca34bee7d0fdcd70694: preserve timestamps in cameraMapper.backup() * b2491ef60e5e23afa7d9f0297f257e694aa1af35: Only attempt to update Wcs if it's available * 9f62bcce588fa9abc8e1e44ff2f0275e5230f629: Registry: hold registry cache for a single thread only (HSC-1035) * 412f03b95b7a5e82003ab33a61bd43adbf465188: Registry: use a pool of registries to avoid having too many open files",2 DM-5284,"Port HSC meas_extensions_simpleShape package to LSST","HSC uses a package, meas_extensions_simpleShape, which needs to be ported to LSST. The package is used for basic shape measurements for determining focus, and also serves as a simple guide for writing measurement plugins.",3 DM-5285,"Port HSC meas_extensions_multiShapelet changesets to LSST","We identified in DM-5162 a few changesets that still need to be ported from HSC to LSST: * bf5f753133ae4b41357f9789ff4763949ebb6ffb: FitPsf: reduce outerOrder to 1 * a54d6cbd41baf916fac2a1bb235a8502af14edfd: Provide explicit instantiations for the sake of clang 6.0 on os/x 10.9 * a53ac9e5cdb678a3f8ef633110d7d4cc5ac84f15: FitProfileAlgorithm: bail if the PSF flux failed ",1 DM-5286,"Port HSC meas_deblender changesets to LSST","We identified in DM-5162 a few changesets that still need to be ported from HSC to LSST: * a8cf6c22df14494d6dcf2d7354c695cba9506301: Clarify tiny footprint limit * 624790aa63a38fb7a328ebc21abfd1b10503aa26: config: change default strayFluxRule * db7d705de93b43a5f32f771c716b1c5c7368d124: consolidate failed peak logic and downgrade warning We also identified a few differences that should be resolved: * clipStrayFluxFraction defaults to 0.01 for LSST, 0.001 for HSC * Stray file, src/Baseline.cc.orig, on LSST side ",1 DM-5287,"Port HSC ip_isr changesets to LSST","We identified in DM-5162 some changesets that still need to be ported from HSC to LSST: * f1cee734998f1faf86c02af42ea599b077847eeb: IsrTask: allow fallback to a different filter when loading calibrations * 89cd629bb8e1a72a545176311b1ef659358d95af: saturationDetection: apply to overscan as well as image ",1 DM-5288,"Port HSC pipe_tasks changesets to LSST","We identified in DM-5162 some changesets that still need to be ported from HSC to LSST: * 31ab5f02f7722650ad0a0eb4e2f7f8b3e0073366, 0c9a4a06bfb34ed26c72109131ef9f4a8c8f237a: multiBand: save background-subtracted coadd as deepCoadd_calexp * e99e140feafe28e6f034143e8ee2ae58e9a9358d: Rejig interface for DetectCoaddSourcesTask to provide non-dataRef-centric API * 829ee0cdd605ed027af1fada4446b715d9a5180d: multiband: activate sky objects * MeasureMergedCoaddSources.doMatchSources defaults to False * ProcessImageConfig.doWriteHeavyFootprintsInSources defaults to False ? * 56666e8feba6893ac95fd4982d3e0daf6baf2d34: WcsSelectImagesTask: catch imagePoly is None We also noticed some differences: * * CalibrateConfig.setDefaults doesn't call parent * CalibrateTask.run isn't returning apCorrMap * reserveFraction=-1 instead of 0.2 ",3 DM-5289,"Port HSC obs_subaru changesets to LSST","We identified in DM-5162 several changesets that still need to be ported from HSC to LSST: * 8948917de4579e032c7bbb2c8316014446e3841b: config: add astrometry filter map for HSC narrow-band filters * 69d35a890234e37c1142ddbeff43e62fe36e6c45: Set radius for flux.naive, adjust comment for flux.sinc * 8ea54d10f5ae56f8b6f244bca76d5796ae015216: config: disable sigma clipping in coadd assembly * 8d2f4a02d0d668fc82e853b633444d8e0fe80010: config: reduce coadd subregionSize * e36bd1b4410812ca314f50c01f899d92acc0e7a5: config: set pixelScale for jacobian correction * Remove processCcdOnsiteDb.py, processStack.py * Rename stacker.py to coaddDriver.py or whatever Nate chooses in DM-3369 * 49e9f5dcf16490f6be6438b89b17911a0cd35fb2: Fixed obvious errors caused by introducing VignetteConfig * 8948917de4579e032c7bbb2c8316014446e3841b: config: add astrometry filter map for HSC narrow-band filters * daa43eeac46e8708de6f37feeb5d5d16a3caca11: HscMapper: set unit exposure time for dark * 77ff7c89d56bed94bca4f320f839dbd20fbab641: Set BAD mask for dead amps instead of SAT We also noticed the following need to be done: * Forced photometry configuration (CCDs and Coadds) * Sanitize config of OBS_SUBARU_DIR (use getPackageDir) * multiband config files need ""root"" --> ""config"" * No astrometry in measureCoaddSources * Narrow bands missing from priority list * detectCoaddSources removed from multiband * Move filterMap from config/processCcd.py into own file",5 DM-5290,"Add z-index for dialogs components","Some of the outside modules that we have brought in have a z-index. We need to make sure that our dialog components stay on top of them.",2 DM-5291,"Docker-ready configuration system for LTD Keeper","To deploy LTD Keeper in a Docker container (DM-5194), it’s best practice to handle all configurations through environment variables. In DM-4950, LTD Keeper was configured through files for test and dev/deployment profiles. What we should do is continue to allow hard-coded configurations for test and dev environments, but have a third fully fledged configuration environment that’s driven entirely by environment variables. The environment variables should allow fine grained configuration (for example, to turn off calls to individual external services for testing). This should also resolve how to deal with Google Container Engine/Kubernetes auth flow works with environment variables, config files, and profiles.",1 DM-5292,"Add queue support","The ctrl_events package currently only supports ActiveMQ topics. This change will add support for queues. Additionally, there will be some minor code and assertion clean up as noted in DM-5279.",12 DM-5294,"ImageDifferenceTask: Refactor Image DifferenceTask","The original DM-3704 was to refactor all ImageDifference task. This issue was split into 3 tasks: 1) Split image difference task into two tasks (1) to generate an image difference, and (2) to run detection and measurement on it: processDiffim.py 2) Refactor the Image Difference portion 3) Refactor the processDiffim portion This ticket refactors the new task that just generates and image difference. ",8 DM-5296,"Rewrite unit tests for new dipole measurement task",NULL,8 DM-5297,"Make jointcal buildable under CI","Once jointcal is part of the stack, we need to get it under continuous integration, buildable by Jenkins, etc. There is only one unittest in the package currently, but at least getting that test built and run will catch a number of basic problems. This requires having its dependencies (CHOLMOD from SuiteSparse) under CI as well. As part of this, it would be good to have at least one ""integration test"" that runs jointcal as part of processCcd, to catch problems that appear when that interface changes.",8 DM-5298,"Document simple simulator","Document the simple simulator produced in DM-4899. This will also involve some refactoring and adding unit tests to make it usable by others in the group.",8 DM-5300,"X16 Design Discussions","Design discussions within the team and with other teams to ensure the plan is complete and properly scoped",64 DM-5301,"Compare LSST and HSC pipelines through through single-frame processing","Following DM-2984, we are confident that the ISR performed by the LSST and HSC stacks is equivalent. Extend that work to cover the whole of single-frame processing ({{ProcessCcdTask}}). There are two deliverables for this issue: * A collections of plots and/or comparison scripts that run on the pipeline outputs that can be used to compare their quality in all the ways we think matter. This should probably be in a new repo. Some of these tests should compare the results of the two pipelines against each other (as that's easier to do), but others should be useful for tracking the quality of the pipeline even after we abandon the HSC fork. * A set of new JIRA issues that capture the code transfers we think we will need to get the LSST pipeline up to the same level of quality as the HSC one.",10 DM-5302,"manage jenkins core + plugin versions","There have been a couple of issues that have arisen when deploying test instances vs updating an existing instance due to slight differences between plugin versions. This would be avoided by putting all plugin versions under change control. Including: * The versions of all jenkins components need to be explicitly specified * The stored job {{config.xml}}'s should be updated to reflect plugin version changes * The hipchat notification configuration should be updated to fix breakage caused by the production core/plugin update earlier this week ",5 DM-5307,"Get high volume test script working again at IN2P3 cluster","Currently runquerys.py script falls over when running high-volume tests: * ""LV"": 75, * ""FTSObj"": 3, * ""FTSSrc"": 1, * ""FTSFSrc"": 1, * ""joinObjSrc"": 1, * ""joinObjFSrc"": 1, * ""nearN"": 1 We need this to be working again to validate recent work on schedulers and to support upcoming work on large results, etc.",10 DM-5308,"Test removal of response queuing on czar to see if this provides useful flow control",NULL,3 DM-5311,"Enable automated publication of qserv-dev release","This would allow integration tests in CI not to break when some Qserv dependencies change. Indeed CI uses a Docker container which include qserv-dev to build current Qserv version.",4 DM-5312,"Additional vertical partitioning tests","Test potential improvements in many-vertical-shards test (20,50) run-times with query optimizer settings.",5 DM-5313,"Refine MemManReal implementation per design discussion w/ John",NULL,10 DM-5314,"Implement unique query-id generation","There are currently two separate query IDs defined for queries in czar code: - ""user query ID"" - defined in {{Czar::submitQuery()}}, used for constructing table names for result table and message table - ""QMeta query ID"" - ID obtained from QMeta after registering the query (by {{UserQuerySelect::_qMetaRegister()}}) Currently user query ID is used by the rest of the czar code to track the processing of this query, QMeta ID is not used yet for anything except QMeta registration and updates. QMeta ID will be used for async query identification and there is no actual reason to keep two IDs around, so we should replace user query ID with the QMeta-generated one everywhere. One minor issue is that currently message table name is built and table is locked before we register query in QMeta. Need to understand it and see if we can reverse that logic.",4 DM-5315,"DM Verificational Plan Document for CoDR","This epic captures work resulting from a Systems Engineering request for a document on the DM Verification plan. In summary, this document must describe how are DM deliverables are going to be verified against the requirements? George agrees that structuring this DM Verification Plan around the already drawn KPMs is the right thing to do. SQuaRE will draft a document for internal DM circulation and eventually leading to a project-led Conceptual Design Review of the DM Verification Plan. The skeleton plan is: * Go through the KPMs * List the method (== tools) by which is KPM will be measured => Describe the requirements for that method => Criteria of success * State when it can be measured * Describe the data necessary or planned for doing the measurements Additionally, a process (most likely an end2end run) will be described that can verify that external to DM interfaces are being correctly met. There is no requirement from the point of the CoDR for a resource-loaded plan leading to this work. That is expected to follow from a successful CoDr. ",28 DM-5316,"Research alert production database design",NULL,4 DM-5317,"Identify specs within VO stack which should be implemented by database team",NULL,6 DM-5318,"Begin exploratory TAP implementation within dbserv","This is a quick coding foray, to try to shake loose unforseen implementation dependencies or speed-bumps with TAP integration. Time-boxed at 4 points to fit into a single sprint with Brian's current resource loading -- this is intended to be only a clarifying start.",4 DM-5319,"Fix mariadb CI","patch package is missing in docker container used by travis-CI.",1 DM-5320,"Make Bright Object Masks compatible with all cameras","Currently all of the logic that goes into using bright object masks falls into obs_subaru and pipe_tasks. This ticket should move parts (such as the bright object mask class) out of obs_subaru, into a camera agnostic location. The work should also duplicate relevant camera configurations and parameter overrides in the other camera packages. Bright object masks were originally introduced in DM-4831",2 DM-5321,"MeasureApCorrTask should use slot_CalibFlux as default ref flux","{{MeasureApCorrTask}} uses ""base_CircularApertureFlux_17_0"" as its default reference flux. It should use ""slot_CalibFlux"" instead. Also check obs_sdss packages for overrides that can be removed; obs_sdss certainly has one in {{config/processCcdTask.py}}",1 DM-5322,"Remove any redundant or unused datasets","Please remove any redundant or unused dataset names from policy files throughout the stack.",1 DM-5323,"estimateBackground should not make a deep copy of the exposure","Implement RFC-155: change {{estimateBackground}} as follows: - Always subtract the background - Modify the exposure in place - Replace {{estimateBackground}} with the run method of a new task {{SubtractBackgroundTask}} - Replace {{getBackground}} (which fits a background) with {{SubtractBackgroundTask.fitBackground}}",4 DM-5324,"Convert GWT code to pure JavaScript (X16, part2, basic)","Continue to work on the GWT code conversion to JavaScript.",100 DM-5327,"Create network monitoring dashboard for nebula sys admins",NULL,4 DM-5328,"JTM meeting in Santa Cruz","Met with many individuals, had lots of good conversations. Helped bring me closer to the working of the project and provided a level-set for where activities are currently at. Attended LHN session for most of Wednesday. ",10 DM-5329,"Assist in OSX VM environment deployment",NULL,2 DM-5330,"Add ExposureIdInfo class","Implement RFC-146: add ExposureIdInfo class to daf_butlerUtils This will be implemented in daf_butlerUtils as part of DM-4692, with a unit test in obs_test because daf_butlerUtils has no camera mapper or camera repo in its test directory.",4 DM-5331,"Add usesMatches to star selectors","Implement RFC-126 add usesMatches to star selectors This will be implemented as part of DM-4692",4 DM-5333,"a Catch all bug fix epic (X16)","A epic for reported bugs in this scycle",10 DM-5334,"GWT Conversion: Table results container","Create a result container for table data. This task is composed of: - create actions, action creators and reducing functions - dynamically add/remove table from view - support expanded mode - TabPanel support for deleting tabs.",6 DM-5336,"Fix minor issues in docker procedure","- params.sh was missing at configuration - startup.py wasn't importing correctly module ""utils"" - remove unused parameters in params.sh",1 DM-5337,"Planning for GPFS, etc.","* Gathered filesize statistics from existing NFS for planning GPFS * Assisted with GPFS client setup on test servers * Reviewed infrastructure changes for Jason",2 DM-5338,"Week end 2/07/16","Support for lsst-dev cluster, OpenStack, and accounts for week ending February 7, 2016.",5 DM-5339,"Week end 2/14/16","Support for lsst-dev cluster, OpenStack, and accounts for week ending February 14, 2016.",6 DM-5340,"Week end 2/21/16","Support for lsst-dev cluster, OpenStack, and accounts for week ending February 21, 2016.",2 DM-5341,"Week end 2/28/16","Support for lsst-dev cluster, OpenStack, and accounts for week ending February 28, 2016. ",4 DM-5342,"Jason Feb Tasks","Procurement activities to prepare ""procurement plan activity 1"". PDU, rack, network selection and review. Refresh quotes for compute, storage, rack, pdu, electrical. Refresh, finalize, present (internally) and review design for FY16 infrastructure. Draft and review of Procurement Plan Activity 1 document.",6 DM-5343,"New equipment setup and configuration (week end 2/07/16)","* Updated lsst-dev7 with few missing pieces after initial user testing * Setup 3 of 8 lsst-test servers * Confirmed IPMI setup on new test servers (working with Dell on issue with 1 iDRAC license upgrade) ** Completed and verified IPMI setups *** Installed licenses for lsst-test1 - lsst-test6 *** re-associated IPMI lsst-test1m last-tsst6m with the correct systems. *** Installed CentOS7 on lsst-test1 - lsst-test6. (in progress) * UPS setup ** Setup table with location of systems in 3003 racks ** Setup apcusbd on lsst-stor141, lsst-stor142, lsst-stor143, lsst-stor144, lsst20, lsst13",3 DM-5344,"Jason Mar Tasks","Week 1: Admin mtg, group mtg, ICI mtg, interview. 1pt Week 2: Meetings, ICI task planning and prioritization 2pt Weeks 3&4: Interviews, admin and group meetings, ARI SOW",6 DM-5345,"New equipment setup and configuration (week end 2/14/16)","* Still pushing at Dell to fix broken iDRAC license * Added 5 systems to RSA OTP system * Completed the setup of lsst-test1, lsst-test4, lsst-test5, lsst-test6 ** Reinstalled lsst-test1 to correct error in puppet install, Completed CentOS7 install, Installed puppet and ran puppet ** Corrected network error on lsst-test4, Completed CentOS7 install, Installed puppet and ran puppet ** Completed the setup of Installed lsst-test5, Completed CentOS7 install, Installed puppet and ran puppet ** Installed lsst-test6, Completed CentOS7 install, Installed puppet and ran puppet",1 DM-5346,"Recover from accumulated technical debt","Through the S15 and W16 cycles the DRP group focused on merging functionality from HSC. To expedite this process, we accepted lower quality documentation and poorer test quality than would normally be required. We need to recover from this, and other, accumulated technical debt.",50 DM-5347,"Add tests for recent improvements to CModel","In DM-4768 we ported a number of improvements to CModel from HSC. However, these were not accompanied by test cases. Please add them.",3 DM-5348,"Get rid of ProcessCcdSdssTask and ProcessCcdDecamTask","Update {{ProcessCcdTask}} so that it can be used with different datasete types as appropriate for the ISR task. This will allow us to get rid of obs-specific variants {{ProcessCcdSdssTask}} and {{ProcessCcdDecamTask}} The plan is to change {{ProcessCcdTask}} as follows: - set {{doMakeDataRefList=False}} in the call to {{add_id_argument}} - get the dataset type from the ISR task (default to ""raw"") and set it in data container - make the dataRef list by calling {{makeDataRefList}} on the data container Question for DECam folks: do you want two executable scripts for DECam (one that processes data from the community pipeline and one that performs ISR)? Or do you prefer one exectutable (in which case you switch between performing ISR and reading the output of the community pipeline output by retargeting the ISR task)? If you prefer one binary, then which should be the default: perform ISR or read the output of the community pipeline?",2 DM-5349,"Revise LSE-140 to account for recent changes to calibration instrumentation","Produce a revision of LSE-140, the DM - to - auxiliary instrumentation ICD, taking into account recent changes to the calibration instrumentation.",5 DM-5350,"Establish goals and create EA framework for LSE-140 update","Deliverable: together with [~pingraham], identify the changes needed and develop initial content in EA.",2 DM-5351,"Create change request for LSE-140","Deliverable: change request and document diffs for LSE-140",1 DM-5352,"Add Vis toolbar to expanded mode",NULL,1 DM-5353,"Upgrade minuit2","Minuit2 5.34.14 came out in 2014. The current version in the stack is 5.28 from 2010. Minuit2 is annotated on the DM third party software page as being approved for 6-monthly uprev. Minuit2 is only used by AFW. Release notes for 5.34.14: * Several fixes and improvements have been put between this verion and the previous stand-alone one (5.28). Main new features is now the support for using the {{ROOT::Math::Minimizer}} interface via the class {{ROOT::Math::Minuit2Minimizer}} also in the standalone version. A new test has been added ({{test/MnSim/demoMinimizer.cxx}}) to show this new functionality * Other major improvements is in the control of the error messages. One can now use the class {{MnPrint::SetLevel(int value)}} to control the output level. The same can be achieved by calling {{ROOT::Math::Minuit2Minimizer::SetPrintLevel}}.",1 DM-5355,"meas_algorithms uses packages that are not listed in table file","{{meas_algorithms}} directly uses the following packages not expressed in the table file: * Minuit2 * daf_persistence * daf_base * pex_config * pex_exceptions * pex_policy ",1 DM-5356,"Test consistency of Shear Measurements with different Psfs","DM-1136 was done with a single Psf, partly to avoid some of the problems we found with PsfShapeletApprox. In this issue, I will look at consistency of the measurement for different Psfs.",8 DM-5357,"Test error estimation with bootstrap resampling","Test the error estimation code using bootstrap resampling.",6 DM-5358,"Move supertask code our from pipe_base","Create a new package {{pipe_supertask}} and move all supertask code and activator there. Will soon create a poll to pick a better name.",3 DM-5359,"Update DMTN-002 to reflect last changes","Need to update documentation with latest changes on {{pipe_base}}, {{pipe_supertask}} and {{pipe_flow}}",1 DM-5360,"Update {{pipe_flow}}","Update {{pipe_flow}} to change dependencies and examples to reflect migration to {{pipe_supertask}}",1 DM-5361,"Begin Image Select Panel/Dialog","This is the first stop in image select panel: * Design basic panel * Implement tabs: issa, 2mass, wise * implement ability to replace a image",10 DM-5362,"Image Select panel: finish tabs","Implement * dss, sdss, blank image tabs * a reusable radius input field * implement upload file tab ( the reusable upload file component is done in DM-5584) ",10 DM-5363,"Image Select Panel: 3 color support","Add 3 color support. Basically take the panel an be able to repeat of red,green, and blue. We might want to use a disclosure component. What has been done: (copied from github commit message) Add 3 colors support to image selection dropdown, disclosure component is used for r, g, b field group tabs. For SizeInpuFiield component, fixed validation, add props 'showFeedback' for feedback display and add valid range in error message popup. (this update is based on the following issues as SizeInputField is used in other panel - Popup message should have the range are part of the message - The feedback at the bottom should be optional, turned on by a property. - When a value such as ‘111d’ is entered, It does not validate correctly. ) Some Issues: - should 'disclosure' component's status be kept? - image creation doesn't work if any of r, g, b is disabled.",10 DM-5364,"Image Select Panel: Support add or modify of plot","previously the image select panel would only modify a plot. Now give it the ability to add a plot.",8 DM-5365,"Enable CC-IN2P3/Qserv team communication in order to prepare for Pan-STARRS large scale tests","The goal of this ticket is to enable communication between CC-IN2P3 and Qserv team in order to prepare for Pan-STARRS data ingestion into Qserv. This data ingestion step is necessary for the large scale tests of Qserv foreseen for summer 2016. Specifically, we need to understand: - What is the size of the data set to be imported to CC-IN2P3? - Where the Pan-STARRS data set to be imported is currently located? - What mechanisms will the host of Pan-STARRS data make available to CC-IN2P3 for downloading the data set? - Does the envisaged ingestion mechanism into Qserv requires that the data transit through the Qserv master server or will each Qserv worker be able to ingest its own chunk of data? - After the ingestion process is finished, do we need to keep a copy of the ingested data out of Qserv? Given the size of the dataset likely involved in this process, this project will probably require that we (both Qserv and CC-IN2P3 experts) set up specific mechanisms and equipment for efficient transport, storage and ingestion of these data. Timely planning and several testing campaigns seem necessary for this project to make progress. ",8 DM-5367,"Change default value of MeasApCorrConfig.refFluxName to slot_CalibFlux","The default value of {{MeasApCorrConfig.refFluxName}} is presently ""base_CircularApertureFlux_17_0"". This should be changed to ""slot_CalibFlux"". That is what the slot is intended for. The slot usually points to ""base_CircularApertureFlux_17_0"", but {{obs_sdss}}, at least, overrides this. Additional jobs: - Update {{obs_sdss}} {{config/processCcd.py}} to remove the override for this value, since it will no longer be needed. - Check for and remove unnecessary overrides in other obs_ packages",1 DM-5368,"Use modern TAP package declarations for all EUPS third party packages","In DM-4670 the TAP-ness of the packages was declared using a {{.tap_package}} file. The modern fix is to use a {{$TAP_PACKAGE}} environment variable in the {{eupspkg.cfg.sh}} file. This is how {{pyyaml}} was implemented.",1 DM-5370,"Create {{lsst_ci}} package as a continuous integration build target","Create an {{lsst_ci}} package to be built for the continuous integration testing. Plan: 1. Create empty package that has dependencies on {{obs_cfht}}, {{obs_decam}}, {{obs_subaru}}, {{testdata_cfht}}, {{testdata_decam}}, {{testdata_subaru}}. (/) 2. Ensure above builds. (/) 3. Add {{obs_lsstSim}} and ensure that it builds. (/) The following were moved to DM-5381: [ [~tjenness] : How can I get strikethrough to work in the following list?] 3. Add dependencies on {{validation_data_cfht}} and {{validation_data_decam}}, and {{validate_drp}}. 4. Run CFHT, DECam quick examples in {{validate_drp}}. 5. Test for successful running of the above examples. Fail and trigger Jenkins FAILURE message if these examples fail. 6. Check performance of CFHT, DECam runs against reference numbers. Fail if there is a significant regression. 7. Decide how to include {{ci_hsc}}, which currently can take at least 30 minutes to process the image data.--",1 DM-5371,"butler planning for X16",NULL,4 DM-5372,"Fix obs_* packages and ci tests broken by DM-4683","The butler changes in DM-4683, in particular the removal of {{.mapper}} from the interface exposed by a {{Butler}} object, broken {{obs_cfht}}, {{obs_decam}}, and {{ci_hsc}}. This issue will fix those changes, and search for additional broken things. This work is proceeding in conjunction with DM-5370 to test that the CI system, e.g. {{lsst_ci}}, is sensitive to these breakages and fixes.",1 DM-5374,"Add to baseline a dedicated replica of L1 database just for scans","Per RFC-133, users will sometimes need to do full table scan through L1 catalogs, and our baseline does not allow for full scans on the L1 catalog. It'd be good to maintain a replica of L1 for such scans. This story involves changing LDM-141 and adding hardware for the replica. ",4 DM-5379,"Remotely attend JTM 2016 sessions","SSIA. The final hours of the final day were very valuable.",6 DM-5380,"L1 base messaging topology.","Make sequence diagram for Base site messaging and enumerate each message, then provide a narrative description of each message (including logical flow control if applicable) AND an example message payload for the message dictionary. 4 Meetings about message exchange style. Meetings about using queue fanout for return messages from forwarders and distributors, or binding using routing keys. 4 Rapid prototyping of some of these ideas for evaluation. 4",12 DM-5381,"Create {{lsst_qa}} package as a daily build target for regression testing","1. Add dependencies on {{validation_data_cfht}} and {{validation_data_decam}}, and {{validation_data_hsc}}. (/) 2. Add dependency on {{validate_drp}}. (/) 3. Run CFHT, DECam quick examples in {{validate_drp}}. (/) 4. Test for successful running of the above examples. (/) 5. Implement in a testing framework. 6. Check performance of CFHT, DECam runs against reference numbers. Fail if there is a significant regression. (/) 7. Include {{ci_hsc}}, which currently takes 1000 seconds to process the image data. 8. Check performance of HSC runs against reference numbers. Fail if there is a significant regression.",4 DM-5384,"Port SdssShape changes from HSC meas_algorithms to LSST meas_base","In porting {{meas_algorithm}} changes from HSC to LSST, modifications to the {{SdssShape}} algorithm were discovered. These changes should be transferred to LSST.",3 DM-5385,"calib_psfReserved is only defined when candidate reservation is activated","The schema should in general not be a function of whether particular features are enabled or disabled so that users can have confidence looking for columns. However, {{MeasurePsfTask}} only creates the {{calib_psfReserved}} column when {{reserveFraction > 0}}. This causes warnings when attempting to propagate flags from calibration catalogs to deep catalogs.",1 DM-5387,"Filter editor","A dialog to edit all the filters on the data for table and XY plot. AND, OR conditions? implemented: * display column's units and descriptions * add single column filter with auto-correction * add free-hand filters field with validation and auto-correction * reset, clear filters as well. * 'Column' is sticky... scrolling left/right will not affect it. ",8 DM-5388,"catalog search panel","The search panel to do the catalog search",10 DM-5389,"GWT Conversion: Dropdown Container","Create drop down container to display search panel, catalog search panel, image search panel, etc. ",4 DM-5390,"JavaScript loading/caching plan","We need to ensure that the latest version of the application(javascript) is loaded. Conditions: 1. once loaded, it should be cached by the browser. 2. name of the script has to be a static, so it can be referenced by api user. 3. it also has to load dependencies(gwt scripts) after the main script is loaded. To do this, we created a tiny firefly_loader.js script whose role is to load the main script and then its dependencies. firefly_loader.js is configured to never cache so that the latest main script is always picked up. The main script is appended with a unique hash on every build. This ensures that the browser will pick up the new script the very first time, and then cache it for future use. ",2 DM-5392,"Please stop leaving repoCfg.yaml files around","After a recent change to {{daf_persistence}} and possibly other packages I'm finding that many packages leave {{repoCfg.yaml}} files lying around after they run unit tests. I'm not sure what is best to do about these files. If they are temporary, as I am guessing, then I think we need some way to clean them up when the tests that generated them have run. If they are intended to be permanent (which would be surprising for auto-generated files) then they should probably be committed? I hope we can do better than adding them to .gitignore.",1 DM-5394,"Investigate boost compiler warnings and update boost to v1.60","As reported in comments in DM-1304 clang now triggers many warnings with Boost v1.59: {code} /Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/boost/1.59.lsst5/include/boost/archive/detail/check.hpp:148:5: warning: unused typedef 'STATIC_WARNING_LINE148' [-Wunused-local-typedef] BOOST_STATIC_WARNING(typex::value); ^ /Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/boost/1.59.lsst5/include/boost/serialization/static_warning.hpp:100:33: note: expanded from macro 'BOOST_STATIC_WARNING' #define BOOST_STATIC_WARNING(B) BOOST_SERIALIZATION_BSW(B, __LINE__) ^ /Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/boost/1.59.lsst5/include/boost/serialization/static_warning.hpp:99:7: note: expanded from macro 'BOOST_SERIALIZATION_BSW' > BOOST_JOIN(STATIC_WARNING_LINE, L) BOOST_STATIC_ASSERT_UNUSED_ATTRIBUTE; ^ /Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/boost/1.59.lsst5/include/boost/config/suffix.hpp:544:28: note: expanded from macro 'BOOST_JOIN' #define BOOST_JOIN( X, Y ) BOOST_DO_JOIN( X, Y ) ^ /Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/boost/1.59.lsst5/include/boost/config/suffix.hpp:545:31: note: expanded from macro 'BOOST_DO_JOIN' #define BOOST_DO_JOIN( X, Y ) BOOST_DO_JOIN2(X,Y) ^ /Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/boost/1.59.lsst5/include/boost/config/suffix.hpp:546:32: note: expanded from macro 'BOOST_DO_JOIN2' #define BOOST_DO_JOIN2( X, Y ) X##Y ^ <scratch space>:25:1: note: expanded from here STATIC_WARNING_LINE148 ^ {code} v1.60 is the current version so we should see if these warnings have been fixed in that version.",2 DM-5395,"Prototype afw/AstroPy integration","Investigate and prototype options for integrating {{afw}} with AstroPy. In particular, this epic focuses on establishing how tightly, if at all, AstroPy and {{afw::table}} should be coupled.",25 DM-5396,"Transfer relevant HSC documentation to LSST","Audit the HSC [questions site|http://hsca.ipmu.jp:8080/questions//] and [Pipeline How To|http://hsca.ipmu.jp/hscsphinx/] documentation. Identify the parts which are still relevant to LSST. Transfer them to the LSST documentation. First assumption is that questions go to [clo|http://community.lsst.org] in an appropriate category, while the howto documentation is incorporated into the developer or user guides as appropriate. Confirm this with SQuaRE before starting work. The intention is to carry out most of this work as a group in a ""dockathon"" session.",20 DM-5397,"X16 Framework bucket","Catch all epic for emergent work in 02C.04.01.",20 DM-5398,"Support for DM replanning process","Throughout the X16 cycle we expect to have to assign effort to support the ongoing DM replanning process. This work takes two forms: * Tasks assigned by the DMLT Working Groups; * Face-to-face discussions with other parts of the DM project. Both are captured in this epic.",25 DM-5399,"HSC port: data release verification","Throughout S15 and W16 we have worked to merge changes from the Hyper Suprime-Cam stack to LSST. This work is now broadly complete, but requires acceptance testing by HSC. In support of that, the LSST stack must be brought to a level at which it is capable of reproducing the January 2016 HSC data release. This work is a continuation of the effort undertaken in DM-3628. It will reach a successful conclusion when the HSC project undertakes future development based on the LSST stack.",50 DM-5400,"Cleanup jointcal","Before we start digging into jointcal, it'd be good to get the whitespace/oldpython/indentation/lint/etc. questions sorted out. This ticket is for that.",2 DM-5401,"Calibration Products Pipeline development in X16","Continued investigation and characterization of the DECam CBP data.",25 DM-5402,"Make cluster deployment scripts more generic and enable ccqserv100...124","These scripts will be improved (i.e. more genericity) and integrated inside Qserv code. Qserv will be deployed on ccqserv100 to ccqserv125",3 DM-5403,"Developer Guide Content & Maintenance Backlog Epic","General maintenance and original content for the DM Developer Guide (http://developer.lsst.io) based on needs during the cycle.",7 DM-5404,"LSST the Docs Production Deployment","In DM-1139 we developed LSST the Docs. LSST the Docs is described in [SQR-006|http://sqr-006.lsst.io]. This epic will focus on the deployment of LSST the Docs as a reliable production service for documentation builds and hosting.",16 DM-5405,"Re-enable CModel forced measurement on CCDs","Recent changes from the HSC side (DM-4768) were implemented in a hurry, and break CModel forced measurement when the reference WCS is different from the measurement WCS (as is the case with forced measurement on CCDs). This was considered an acceptable temporarily, since forced CCD measurement is currently severely limited by our lack of deblending, but we'll need to fix it eventually. The fix is trivial from an algorithmic standpoint but may require a bit of refactoring (at least changing some function signatures; maybe more). This should include re-enabling the different-WCS complexity in testCModelPlugins.py,",2 DM-5406,"Require fields listed in icSourceFieldsToCopy to be present","{{CalibrateTask}} presently treats config field {{icSourceFieldsToCopy}} as a list of fields to copy *if present*. This was required because one of the standard fields to copy was usually missing. However, [~price] fixed that problem in DM-5385. Now we can raise an exception if any field listed is missing (though I propose to continue ignoring {{icSourceFieldsToCopy}} if isSourceCatalog is not provided).",1 DM-5407,"Rename datasets to utilize butler aliases","Now that the butler has alias features that can allow for some degree of dataset substitutability, we should consider renaming (or adding aliases) for our existing datasets to make the naming consistent and analysis code more generic. This work should be proceeded by an RFC with a proposal for the new names and a migration plan. It *might* make sense to defer this until the high-level pipeline descriptions are more mature and we can choose relatively future-proof names, but hopefully the alias features will also make migration easy enough that this doesn't matter a lot.",4 DM-5408,"Qserv do not return very same BLOB field than MySQL","Enabling query {{qserv_testdata/datasets/case01/queries/0007.2_fetchSourceByObjIdSelectBLOB.sql.FIXME}} will reveal this bug. Qserv chunk table contains next BLOB: {code:bash} mysql --socket /home/dev/qserv-run/git/var/lib/mysql/mysql.sock --user=root --password=changeme qservTest_case01_qserv -e ""select blobField from Source_6630 where SourceId=29809239313746172;"" > 29809239313746172.chunk6630 vi 29809239313746172.chunk6630 blobField ^D^B\0^W\0\0\0^B\0^K\0~B\0first_fieldsecond_field(�q.\\� {code} But Qserv returns: {code} ^D^B\0^W\0\0\0^B\0^K\0~B\0first_fieldsecond_field(�q.� {code} See DM-991 for additional informations. ",8 DM-5410,"DecamIngestTask is mis-calling openRegistry","`DecamIngestTask` is mis-calling `lsst.pipe.tasks.RegistryTask`. Line 59: {code} with self.register.openRegistry(args.butler, create=args.create, dryrun=args.dryrun) as registry: {code} {{openRegistry}} is expecting a directory name, not a butler object for the first argument Thanks to [~wmwood-vasey] for diagnosing this.",1 DM-5412,"Test new dipole fitting task on real data","Test new task on real data (which data, TBD); inspect results by eye and compare with existing DipoleMeasurementTask output. This is necessary prior to incorporation into the imageDifference command-line task. This test may also indicate that further optimizations are necessary (DM-5721).",6 DM-5413,"Incorporate new DipoleFitTask into imageDifference command-line task alongside existing DipoleMeasurementTask","Incorporate the new task into the command-line task. The goal of this ticket is to implement DipoleFitTask along-side the existing DipoleMeasurementTask, eventually to replace it. This is likely to have additional stories added, including testing, possibly as part of DM-5412.",6 DM-5414,"Create buildable SuiteSparse external package","To get jointcal to build in the stack, we need to satisfy the SuiteSparse dependency by creating an external package for SuiteSparse. Assuming it builds cleanly, this should satisfy the remaining requirement of RFC-153, now that the licensing question has been answered there.",2 DM-5415,"generation of conda binary packages for DM software products","This epic covers work in generating binaries for stack releases. At this point we are persisting with the plan to produce conda binary packages for ease of use on the user side, though their reliable generation has so far resisted automation. Conda binaries will be produced for the 12.0 Stack release. ",32 DM-5416,"Ci Deploy and Distribution Improvements part IV","This is a bucket epic for ongoing improvements to the CI system",8 DM-5418,"Release engineering Part Three","This epic covers testing and co-ordination work associated with making engineering and official releases, and code to support them. (FE:8, DN:6.5, JS:8)",23 DM-5419,"ci_hsc fails test requiring >95% of PSF stars to be stars on the coadd","Since the first week of March 2016, ci_hsc fails its test that requires that >95% of the PSF stars be identified as stars in the coadd. I suspect this is related to the DM-4692 merge. Here is a sample job that fails: https://ci.lsst.codes/job/stack-os-matrix/9084/label=centos-6/console The relevant snippet of the failure is: {code} [2016-03-10T17:12:06.667778Z] : Validating dataset measureCoaddSources_config for {'filter': 'HSC-R', 'tract': 0, 'patch': '5,4'} [2016-03-10T17:12:06.697383Z] CameraMapper: Loading registry registry from /home/build0/lsstsw/build/ci_hsc/DATA/registry.sqlite3 [2016-03-10T17:12:06.697615Z] CameraMapper: Loading calibRegistry registry from /home/build0/lsstsw/build/ci_hsc/DATA/CALIB/calibRegistry.sqlite3 [2016-03-10T17:12:07.716310Z] CameraMapper: Loading registry registry from /home/build0/lsstsw/build/ci_hsc/DATA/registry.sqlite3 [2016-03-10T17:12:07.716443Z] CameraMapper: Loading calibRegistry registry from /home/build0/lsstsw/build/ci_hsc/DATA/CALIB/calibRegistry.sqlite3 [2016-03-10T17:12:08.663566Z] : measureCoaddSources_config exists: PASS [2016-03-10T17:12:08.721051Z] : measureCoaddSources_config readable (<class 'lsst.pipe.tasks.multiBand.MeasureMergedCoaddSourcesConfig'>): PASS [2016-03-10T17:12:08.721077Z] : Validating dataset measureCoaddSources_metadata for {'filter': 'HSC-R', 'tract': 0, 'patch': '5,4'} [2016-03-10T17:12:08.721249Z] : measureCoaddSources_metadata exists: PASS [2016-03-10T17:12:08.721663Z] : measureCoaddSources_metadata readable (<class 'lsst.daf.base.baseLib.PropertySet'>): PASS [2016-03-10T17:12:08.721715Z] : Validating dataset deepCoadd_meas_schema for {'filter': 'HSC-R', 'tract': 0, 'patch': '5,4'} [2016-03-10T17:12:08.721878Z] : deepCoadd_meas_schema exists: PASS [2016-03-10T17:12:08.726703Z] : deepCoadd_meas_schema readable (<class 'lsst.afw.table.tableLib.SourceCatalog'>): PASS [2016-03-10T17:12:08.726834Z] : Validating source output for {'filter': 'HSC-R', 'tract': 0, 'patch': '5,4'} [2016-03-10T17:12:10.203469Z] : Number of sources (7595 > 100): PASS [2016-03-10T17:12:10.204166Z] : calib_psfCandidate field exists in deepCoadd_meas catalog: PASS [2016-03-10T17:12:10.204772Z] : calib_psfUsed field exists in deepCoadd_meas catalog: PASS [2016-03-10T17:12:10.205468Z] : Aperture correction fields for base_PsfFlux are present.: PASS [2016-03-10T17:12:10.206159Z] : Aperture correction fields for base_GaussianFlux are present.: PASS [2016-03-10T17:12:10.207193Z] FATAL: 95% of sources used to build the PSF are classified as stars on the coadd (0 > 0): FAIL [2016-03-10T17:12:10.207455Z] scons: *** [.scons/measure-HSC-R] AssertionError : Failed test: 95% of sources used to build the PSF are classified as stars on the coadd (0 > 0) [2016-03-10T17:12:10.207481Z] Traceback (most recent call last): [2016-03-10T17:12:10.207525Z] File ""/home/build0/lsstsw/stack/Linux64/scons/2.3.5/lib/scons/SCons/Action.py"", line 1063, in execute [2016-03-10T17:12:10.207556Z] result = self.execfunction(target=target, source=rsources, env=env) [2016-03-10T17:12:10.207593Z] File ""/home/build0/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py"", line 133, in scons [2016-03-10T17:12:10.207611Z] return self.run(*args, **kwargs) [2016-03-10T17:12:10.207646Z] File ""/home/build0/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py"", line 122, in run [2016-03-10T17:12:10.207663Z] self.validateSources(dataId) [2016-03-10T17:12:10.207732Z] File ""/home/build0/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py"", line 191, in validateSources [2016-03-10T17:12:10.207749Z] 0.95*psfStars.sum() [2016-03-10T17:12:10.207786Z] File ""/home/build0/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py"", line 52, in assertGreater [2016-03-10T17:12:10.207816Z] self.assertTrue(description + "" (%d > %d)"" % (num1, num2), num1 > num2) [2016-03-10T17:12:10.207853Z] File ""/home/build0/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py"", line 43, in assertTrue [2016-03-10T17:12:10.207877Z] raise AssertionError(""Failed test: %s"" % description) [2016-03-10T17:12:10.207919Z] AssertionError: Failed test: 95% of sources used to build the PSF are classified as stars on the coadd (0 > 0) [2016-03-10T17:12:10.209935Z] scons: building terminated because of errors. {code} This is the test that fails https://github.com/lsst/ci_hsc/blob/74303a818eb5049a2015b5e885df2781053748c9/python/lsst/ci/hsc/validate.py#L169 {code} class MeasureValidation(Validation): _datasets = [""measureCoaddSources_config"", ""measureCoaddSources_metadata"", ""deepCoadd_meas_schema""] _sourceDataset = ""deepCoadd_meas"" _matchDataset = ""deepCoadd_srcMatch"" def validateSources(self, dataId): catalog = Validation.validateSources(self, dataId) self.assertTrue(""calib_psfCandidate field exists in deepCoadd_meas catalog"", ""calib_psfCandidate"" in catalog.schema) self.assertTrue(""calib_psfUsed field exists in deepCoadd_meas catalog"", ""calib_psfUsed"" in catalog.schema) self.checkApertureCorrections(catalog) # Check that at least 95% of the stars we used to model the PSF end up classified as stars # on the coadd. We certainly need much more purity than that to build good PSF models, but # this should verify that flag propagation, aperture correction, and extendendess are all # running and configured reasonably (but it may not be sensitive enough to detect subtle # bugs). psfStars = catalog.get(""calib_psfUsed"") extStars = catalog.get(""base_ClassificationExtendedness_value"") < 0.5 self.assertGreater( ""95% of sources used to build the PSF are classified as stars on the coadd"", numpy.logical_and(extStars, psfStars).sum(), 0.95*psfStars.sum() ) {code} Note that the assertion failure messages is a bit confusing. It should say ""Fewer than 95% of the sources used to build the PSF are classified as stars on the coadd.""",1 DM-5420,"Integration and test monitoring implementation Part I","Configure, develop and deploy an ELK system. High level requirements: * es.lsst.codes - An current version Elasticsearch cluster. * collect.lsst.codes - A server with multiple services to collect and aggregate logging and messages. * logging.lsst.codes - A server with Kibana hooked up between. * Packer and ansible deploys to create artifacts and deploys on Nebula and AWS. Optionally Docker. * Use ELK to monitor git-lfs and our CI system. ",52 DM-5422,"Understand and test real space extension for ZOGY","The ZOGY algorithm (http://arxiv.org/abs/1601.02655) can be implemented as a real space extension of A&L.",43 DM-5423,Documentation,"We are reserving time this cycle for people to contribute to architecture and documentation efforts.",38 DM-5424,"Switch PropagateVisitFlags to use src instead of icSrc","On DM-5084 [~jbosch] switched PropagateVisitFlags to match against icSrc instead of src because we weren't yet matching `icSrc` to `src` in ProcessCcdTask. That's now been done on DM-4692, so we can revert this. After doing so, please verify with ci_hsc that this is working, as that's where the only test of this feature lives.",2 DM-5425,"Provide an easy way to set Coord fields of a source catalog","We sometimes need to set the coord fields of a source catalog, e.g. when fitting a new WCS or when studying an `icSrc` catalog (whose Coord field is not set). It would be nice to have a central, easily found way to do this. Right now we have the following as a static method of `TanSipWcsTask`, which works fine but is in a poor location: {code} def updateSourceCoords(wcs, sourceList): """"""Update coords in a collection of sources, given a WCS """""" if len(sourceList) < 1: return schema = sourceList[1].schema srcCoordKey = afwTable.CoordKey(schema[""coord""]) for src in sourceList: src.set(srcCoordKey, wcs.pixelToSky(src.getCentroid())) {code} The other direction is also useful for reference catalogs, though from a practical standpoint the only user is probably `meas_astrom`. Even so, I suggest that this be made publicly available in the same way. Again, this is presently a static method of `FitTanSipWcsTask`: {code} def updateRefCentroids(wcs, refList): """"""Update centroids in a collection of reference objects, given a WCS """""" if len(refList) < 1: return schema = refList[0].schema coordKey = afwTable.CoordKey(schema[""coord""]) centroidKey = afwTable.Point2DKey(schema[""centroid""]) for refObj in refList: refObj.set(centroidKey, wcs.skyToPixel(refObj.get(coordKey))) {code} I hope this can remain Python code, but admit that the extra speed of C++ might come in handy in some cases. In any case, once the function is in a central location we can implement it in C++ if we find the need.",2 DM-5426,"Participate in X16 DMLT Working Groups","Bosch, Lupton & Swinbank are members of [DMLT Working Groups|https://confluence.lsstcorp.org/display/DM/DMLT+Working+Groups] during X16. This epic captures work related to the activities of those groups.",45 DM-5427,"SingleFrameVariancePlugin can give numpy warnings","SingleFrameVariancePlugin can produce the following numpy warning, with no hint as to where the problem is coming from: {code} /Users/rowen/UW/LSST/lsstsw/miniconda/lib/python2.7/site-packages/numpy/core/_methods.py:59: RuntimeWarning: Mean of empty slice. warnings.warn(""Mean of empty slice."", RuntimeWarning) {code} I tracked it down by adding the following code to the calling code: {code} import warnings with warnings.catch_warnings(): warnings.filterwarnings('error') {code} It would be nice if the measurement plugin handled this situation more gracefully, such as turning the warning into an exception or testing for it and handling it. One way to reproduce this problem is to run {{tests/testProcessCcd.py}} in {{pipe_tasks}}. However, it is commonly seen when running {{processCcd}} on other data, as well.",2 DM-5428,"ObjectSizeStarSelector can produce numpy warnings","`ObjectSizeStarSelector` can produce the following numpy warning: {code} RuntimeWarning: invalid value encountered in less {code} This occurs at the following point in the code: {code} for i in range(nCluster): # Only compute func if some points are available; otherwise, default to NaN. pointsInCluster = (clusterId == i) if numpy.any(pointsInCluster): centers[i] = func(yvec[pointsInCluster]) {code} where `func` has been assigned to `numpy.mean`. When I have seen this occur I have found that `dist` is an array of `nan` I suggest that the star selector handle this situation more gracefully, e.g. by reporting an appropriate exception or handling the data in an appropriate way. If logging a message would be helpful, then please do that (and if RFC-154 is adopted, a log will be available). One way to reproduce this is to run `tests/testProcessCcd.py` in `pipe_tasks`. However, I often see it when running `processCcd.py` on other data, as well.",2 DM-5430,"Tune and improve ngmix MCMC sampling","Improve the ngmix MCMC sampling plugin to get it working well on most sources. This may require actually contacting Erin Sheldon and getting his help (but he's quite eager to help).",10 DM-5431,"Changes to galaxy_shear_experiments Python code","This ticket describes changes which were made to the test runner and analysis scripts during the Dec 2015 - Feb 2016 period. Most of these changes were made as a part of moving to a large computing cluster, where both the units of work and the output file organization had to be changed to make parallelization possible. The large number of tests run during this period and the need to more efficiently analyze and compare also introduced some changed to the analysis and plot modules. Since these changes do not pertain to any single test (though many were done during Dm-1136), I have put them on a separate ticket.",5 DM-5432,"Add SFM plugin for ngmix fitting","Add an SFM pluggin for ngmix fitting using one of the simple fitters in ngmix/fitting.py. This should depend on DM-5429 (or a suitably configured modelfit_ShapeletPsfApprox) for approximating the PSF as a sum of Gaussians. Testing and tuning this algorithm to get it working well should be deferred to another issue.",10 DM-5434,"Ensure that new object loads are added to secondary index","L3 users might be generating new objectIds that are not in the DR Object table. If that is the case, ad-hoc L3 analysis would be triggering updates to secondary index. It is possible that L3 users will be bringing data from other surveys and might partition it DR-way and cross match. We need some mechanism to mark the secondary index dirty when something new gets added to the Object table, and trigger refresh.",10 DM-5435,"Provide a shared stack on lsst-dev & other relevant systems","Following the discussion in RFC-156, ensure that a documented, fast, easy to initialize shared stack is available for developers to use on shared systems, certainly to include {{lsst-dev}}.",3 DM-5436,"Create unit test for ip_isr fallbackfilter","DM-5287 introduced a configuration option that allows specifying a fallback filter in the event that getting a specific butler product fails. Currently there is no test for this functionality. One should be created which tests all the logical paths. This may involve just adapting or mimicking another test that already exists.",2 DM-5437,"Move tests/negative.py from meas_algorithms to meas_base","Porting code from HSC to LSST brought over a unit test into meas_algorithms for functionality that exists in meas_base in LSST. This is due to the refactoring of code into meas_base on the LSST some while ago. This unit test currently runs with code from meas_algorithms, which means it can not simply be moved, as meas_base comes before meas_algorithms in the build order. This work may involve rewriting the unit test to use different code, or evaluating if it is worth bringing that functionality to meas_base along with the test. The code in question is the detection task.",2 DM-5438,"Data Backbone ConOps","Data backbone first edit : 1pt (week 1) Data backbone second edit : 1 pt (week 2) Third edition : 1 pt week 3&4",3 DM-5441,"Astropy integration with LSST DM Software","Work covering the investigation of how to integrate Astropy into the LSST DM software stack.",30 DM-5443,"Compare Astropy and LSST functionality","This story will examine the overlap between Astropy and AFW and examine different approaches that could be taken to integrate Astropy into the DM software.",10 DM-5444,"Write Report on Astropy integration proposals","A report is to be written on the Astropy integration plan. This report will be in the form of an SPIE paper.",20 DM-5445,"Convert GWT code to pure JavaScript (X16, part3 visualization)","This Epic is for the remaining effort in Extra 2016 cycle related to Firefly visualization coed conversion from GWT to pure JavaScript. ",100 DM-5446,"Add scipy as a stack dependency","Adding scipy as a stack dependency is still a nebulous term to me. David is going to follow up on how to do this exactly (it's already in conda_packages.txt).",2 DM-5447,"Write technical note describing galaxy shear fitting experiments","Through S15 (DM-1108) and W16 (DM-3561), [~pgee] has conducted a large-scale investigation into galaxy shear fitting. Please summarize the motivation, methodology and results of this study as a [technical note|http://sqr-000.lsst.io/en/master/].",8 DM-5448,"Familiarization with ngmix codebase","Download the ngmix codebase from https://github.com/esheldon/ngmix. Install it and its dependencies in the same environment as the LSST stack. Experiment with using it and understanding how it works",3 DM-5449,"Convert GWT code to pure JavaScript (F16)","The remaining work for converting GWT code to pure JavaScript",100 DM-5450,"Visualization algorithm related research (S17)","We have some algorithm related issues that need some research time. ",40 DM-5451,"inter team discussion (X16)","This epic is reserved for inter team discussion and supply/collect input to/from other teams.",6 DM-5452,"create support in Butler for multiple repositories","We need to be able to find repositories based on criteria such as version, validity date, etc. This story is to provide support & proof of concept that demonstrates this.",12 DM-5455,"Implement experimental DCR correction","After the discussion about DCR, a few avenues for dealing with DCR were enumerated. It was found that using imaging to model the DCR could be a very fruitful approach. Nate Lust has suggested an algorithm that performs well in simplified, one-dimensional systems. This epic is to extend this algorithm to 2-dimensions and add realistic SEDs, bandpasses, etc. The result will be an implementation of the algorithm applied to the simulated data. With measurements of how well it corrects for DCR in the context of image differencing.",54 DM-5457,"Adapt LTD Mason for Single-package doc builds on Travis CI","LTD Mason was originally intended to build docs for DM’s Eups-based packages from our Jenkins CI/CD servers. There is tremendous value in consolidating all of DM’s Sphinx-based documentation deployments to use LSST the Docs rather than Read the Docs. This ticket will design and implement adaptations to LTD Mason to build single repo doc projects (Technotes, Design Documents, the Developer Guide, and even generic software projects) from a Travis CI environment. Also includes a template {{.travis.yml}} and associated documentation to allow others to enable travis builds for their documentation. We name Travis specifically because it is the easiest platform for implementing CI for generic open source projects.",4 DM-5458,"Update SQR-006 LSST the Docs technote to reflect deployment in DM-5404","This ticket will ensure that [SQR-006|http://sqr-006.lsst.io] reflects the LSST the Docs continuous delivery platform as it is deployed in the DM-5404 epic. (SQR-006 was initially written as a planning/design document). This story should be closed only once the DM-5404 epic is ready to be closed.",4 DM-5462,"Add non-linearity correction to ISR task","Implement RFC-164 At the moment some preliminary code is on ticket branches, but this need to be redone once the RFC is finished.",6 DM-5463,"Don't restore the mask in CharacterizeImageTask.characterize","CharacterizeImageTask.characterize presently restores the mask from a deep copy for each iteration of the loop to compute PSF. This is unnecessary because repair and detection both clear the relevant mask planes before setting new values.",1 DM-5465,"Finish getting obs_decam ISR working with CBP data","Success criteria: * Flats should be totally flat, i.e. bias jump problem fixed everywhere, amplifier levels fixed (both of these are currently hit & miss at the moment). * CRs should be properly interpolated over for non-sky images (as this means no PSF estimate). * Use un-binned images to confirm that bad pixel masks are correct everywhere. ",10 DM-5468,"S17 Qserv Refactoring","Refactoring of Qserv as found necessary in F16. Specific tasks will be added during F16, and will include bug fixes, fixing major deficiencies discovered during F16, and keeping Qserv code up-to-date (latest compilers, supported OSes, security and alike). The scope of the work is limited by the number of story points assigned to this epic. ",23 DM-5470,"Develop C++ code for experimenting with Python binding","Produce a small C++ codebase that can be used for experimenting with the various technologies we can be used for exposing C++ to Python. It should enable us to experiment with as many of the potential pain points with these technologies as possible",3 DM-5471,"Wrap example C++ code with Cython","Take the example C++ codebase developed in DM-5470, and expose it to Python in the most idiomatic possible way using Cython. Produce a [technical note|http://sqr-000.lsst.io] describing how this was carried out and discussing any particular pain points either in implementation or results.",10 DM-5472,"Update meas_mosaic for compatibility with new single frame processing","Following [recent changes to single frame processing|https://community.lsst.org/t/backward-incompatible-changes-to-processccdtask-and-subtasks/581], {{icSrc}} no longer includes celestial coordinates and {{icMatch}} is no longer being written. {{meas_mosaic}} requires this information. Provide a work-around.",3 DM-5473,"Jenkins/ci_hsc failure: 'base_PixelFlags_flag_clipped' already present in schema","Since 15 March, the {{ci_hsc}} build in Jenkins has been failing as follows: {code} [2016-03-16T14:23:13.548928Z] Traceback (most recent call last): [2016-03-16T14:23:13.548956Z] File ""/home/build0/lsstsw/stack/Linux64/pipe_tasks/2016_01.0-23-gcf99090/bin/measureCoaddSources.py"", line 3, in <module> [2016-03-16T14:23:13.548969Z] MeasureMergedCoaddSourcesTask.parseAndRun() [2016-03-16T14:23:13.548999Z] File ""/home/build0/lsstsw/stack/Linux64/pipe_base/2016_01.0-6-g7751869/python/lsst/pipe/base/cmdLineTask.py"", line 450, in parseAndRun [2016-03-16T14:23:13.549011Z] resultList = taskRunner.run(parsedCmd) [2016-03-16T14:23:13.549040Z] File ""/home/build0/lsstsw/stack/Linux64/pipe_base/2016_01.0-6-g7751869/python/lsst/pipe/base/cmdLineTask.py"", line 192, in run [2016-03-16T14:23:13.549048Z] if self.precall(parsedCmd): [2016-03-16T14:23:13.549076Z] File ""/home/build0/lsstsw/stack/Linux64/pipe_base/2016_01.0-6-g7751869/python/lsst/pipe/base/cmdLineTask.py"", line 279, in precall [2016-03-16T14:23:13.549087Z] task = self.makeTask(parsedCmd=parsedCmd) [2016-03-16T14:23:13.549115Z] File ""/home/build0/lsstsw/stack/Linux64/pipe_base/2016_01.0-6-g7751869/python/lsst/pipe/base/cmdLineTask.py"", line 369, in makeTask [2016-03-16T14:23:13.549132Z] return self.TaskClass(config=self.config, log=self.log, butler=butler) [2016-03-16T14:23:13.549160Z] File ""/home/build0/lsstsw/stack/Linux64/pipe_tasks/2016_01.0-23-gcf99090/python/lsst/pipe/tasks/multiBand.py"", line 1008, in __init__ [2016-03-16T14:23:13.549179Z] self.makeSubtask(""measurement"", schema=self.schema, algMetadata=self.algMetadata) [2016-03-16T14:23:13.549206Z] File ""/home/build0/lsstsw/stack/Linux64/pipe_base/2016_01.0-6-g7751869/python/lsst/pipe/base/task.py"", line 226, in makeSubtask [2016-03-16T14:23:13.549846Z] subtask = configurableField.apply(name=name, parentTask=self, **keyArgs) [2016-03-16T14:23:13.549901Z] File ""/home/build0/lsstsw/stack/Linux64/pex_config/2016_01.0+1/python/lsst/pex/config/configurableField.py"", line 77, in apply [2016-03-16T14:23:13.549915Z] return self.target(*args, config=self.value, **kw) [2016-03-16T14:23:13.549943Z] File ""/home/build0/lsstsw/stack/Linux64/meas_base/2016_01.0-12-gf26bc28+1/python/lsst/meas/base/sfm.py"", line 248, in __init__ [2016-03-16T14:23:13.549954Z] self.initializePlugins(schema=self.schema) [2016-03-16T14:23:13.549985Z] File ""/home/build0/lsstsw/stack/Linux64/meas_base/2016_01.0-12-gf26bc28+1/python/lsst/meas/base/baseMeasurement.py"", line 298, in initializePlugins [2016-03-16T14:23:13.550004Z] self.plugins[name] = PluginClass(config, name, metadata=self.algMetadata, **kwds) [2016-03-16T14:23:13.550032Z] File ""/home/build0/lsstsw/stack/Linux64/meas_base/2016_01.0-12-gf26bc28+1/python/lsst/meas/base/wrappers.py"", line 15, in __init__ [2016-03-16T14:23:13.550616Z] self.cpp = self.factory(config, name, schema, metadata) [2016-03-16T14:23:13.550647Z] File ""/home/build0/lsstsw/stack/Linux64/meas_base/2016_01.0-12-gf26bc28+1/python/lsst/meas/base/wrappers.py"", line 223, in factory [2016-03-16T14:23:13.550660Z] return AlgClass(config.makeControl(), name, schema) [2016-03-16T14:23:13.550688Z] File ""/home/build0/lsstsw/stack/Linux64/meas_base/2016_01.0-12-gf26bc28+1/python/lsst/meas/base/baseLib.py"", line 3401, in __init__ [2016-03-16T14:23:13.552891Z] this = _baseLib.new_PixelFlagsAlgorithm(*args) [2016-03-16T14:23:13.552924Z] lsst.pex.exceptions.wrappers.InvalidParameterError: [2016-03-16T14:23:13.552967Z] File ""src/table/Schema.cc"", line 563, in lsst::afw::table::Key<lsst::afw::table::Flag> lsst::afw::table::detail::SchemaImpl::addField(const lsst::afw::table::Field<lsst::afw::table::Flag>&, bool) [2016-03-16T14:23:13.552986Z] Field with name 'base_PixelFlags_flag_clipped' already present in schema. {0} [2016-03-16T14:23:13.553012Z] lsst::pex::exceptions::InvalidParameterError: 'Field with name 'base_PixelFlags_flag_clipped' already present in schema.' [2016-03-16T14:23:13.553014Z] [2016-03-16T14:23:13.613484Z] scons: *** [.scons/measure] Error 1 [2016-03-16T14:23:13.617577Z] scons: building terminated because of errors. {code} Please fix it.",1 DM-5474,"Bugs in obs_subaru found by PyFlakes","I ran pyflakes on the code in obs_subaru and found a few bugs (beyond a few trivial ones that I am fixing as part of DM-5462) {{ingest.py}} has undefined name {{day0}} {{ccdTesting.py}} has at least three undefined variables: {{x}}, {{y}} and {{vig}} in the following: {code} ngood += pupilImage[y[good], x[good]].sum() vig[i] = float(ngood) {code} {{crosstalkYagi.py}} has many undefined names, starting with {{makeList}}, {{estimateCoeffs}}",1 DM-5475,"Document investigation of logging, monitoring and metrics technologies and architecture","Finish technote SQR-007. Related to DM-4970",4 DM-5476,"Revise FlagHandler"," {{FlagHandler}} is ""unpolished ... and a bit dangerous to the unwary"" (DM-5247). It could be improved by leveraging C++11 features, replacing the default constructor with something that defines the (required) general failure flag, and allowing flags to be added individually. A potential starting point is [here|https://jira.lsstcorp.org/browse/DM-5247?focusedCommentId=45894&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-45894].",4 DM-5477,"Firefly support for Camera team visualization needs (X16)","Attend the weekly meeting with the camera team and UIUC development team, provide support in discussion and API usage. ",4 DM-5478,"Write script to derive and collate QA metrics from data repository of processed data","I wrote a python script using stack components to derive QA metrics and collate other QA-relevant information for a data repository of processed data. This is currently output to a CSV file that can be loaded into a SQL database.",20 DM-5479,"Wrote script to print the names of all visits that overlap a patch","In order to finish the IDL workflow module for makeCoaddTempExp I needed a program to say which visits overlap a given path. That's what this script does.",5 DM-5480,"Processing of COSMOS data - Part II","Continued work on processing and QA work on the COSMOS verification dataset. Running processCcDecam, making diagnostic plots, and nvestigating the results. Most recently I've reprocessed the COSMOS data through processCcdDecam using SDSS as the astrometric and photometric reference catalog and am redoing the QA work on those results.",20 DM-5481,"Write software to match up and combine data for sources in a processed data repository","In order to fully check the outputs of the processed COSMOS data I needed to combine the information on sources from multiple visits. This code (written in IDL for now) matching up sources astrometrically across different visits, combines all of the information on separate detections, and measures average quantities (phot and astrom) for unique sources. The information is then output into four binary FITS files.",10 DM-5482,"Write presentation on verification datasets for AAS","Prepared and gave a talk at the NSF booth at the Florida AAS meeting on the progress of the verification datasets effort.",5 DM-5483,"Work on script to test the astrometric matcher","We encouraged astrometric matching problems for the Bulge verification dataset. Therefore, I wrote a script that tests the matcher by systematically shifting the coordinates of one sets of the data to see if the matcher still works. It worked well until ~80 arcsec.",5 DM-5484,"SdssMapper.paf has wrong python type for processCcd_config","[~npease] reports that {{Sdssmapper.paf}} has the wrong python data type for the dataset {{processCcd_config}}: it is {{lsst.obs.sdss.processCcdSdss.ProcessCcdSdssConfig}} instead of {{lsst.pipe.tasks.processCcd.ProcessCcdConfig}}",1 DM-5485,"Work on plan to test specific algorithmic components of the stack","After working on a script to test the astrometric matcher, I decided to put together a plan to run similar tests on our algorithmic code. The rough plan is here: https://confluence.lsstcorp.org/display/SQRE/Stack+Testing+Plan",2 DM-5486,"Work on putting together page of ""tips and tricks for using the stack""","Due to the incomplete state of the stack documentation and tutorials, I decided to write down various ""tips and tricks"" for using the stack as I learn them. https://confluence.lsstcorp.org/display/SQRE/Tips+and+Tricks+for+using+the+Stack",2 DM-5487,"Revise operations concept for Observation Processing System","Turn the L1 ConOps document into appropriate sections of LDM-230, specifying automated operations sequences, how human intervention can occur, and processes to handle changes and updates. (Story points are for KTL drafting and initial contributions)",2 DM-5488,"Field group updates","After some work we have realized that the following needs to be done to field groups: * Tabs group should have a field group smart wrapper component * field group needs to reinit on id change * remove mixin, use Higher-Order Components instead * support a function for a value, this function will return a value or a promise * hidden fields - init field group with key/value object * Sub-field groups? study only, unless it is easy to implement. * maintain an option to keep unmount field value available * determine if InitValue needs to be passed around * passing fieldState around too much * find reason for react warning every time popup is raised * look at promise code make sure it is working the way we think * if practical, remove all export default FieldGroupConnector.  It is the high order component that replaces the mixin. FieldGroupUtils.js: (~line 33): The field value would be a function on the file upload case. Therefore the upload does not activate until validation. In the upload case the function would return a promise. However, It could return a value or an object with a value and a valid status. Now the value key of a field can contain a promise or function or primitive. The function can return a primitive, a promise, or an object with primitive and status. fftools.js lines 102-158 you can see my experimenting with taking out the connector. It works fine and does eliminate one of the warning messages. ",8 DM-5489,"improvement of the north/east arrow on image","make the compass sticky when scroll the image",1 DM-5490,"Develop operations concept for Batch Processing System","Develop a ConOps document that can be included as appropriate sections of LDM-230 describing the batch processing environment, specifying automated operations sequences, how human intervention can occur, and processes to handle changes and updates. (Story points are for KTL drafting and initial contributions)",3 DM-5491,"Develop operations concept for Data Backbone","Develop a ConOps document that can be included as appropriate sections of LDM-230 describing the Data Backbone that contains, manages, and provides access to the Science Data Archive, specifying automated operations sequences, how human intervention can occur, and processes to handle changes and updates. (Story points are for KTL drafting and initial contributions)",3 DM-5492,"Develop operations concept for Data Access Processing System","Develop a ConOps document that can be included as appropriate sections of LDM-230 describing the Data Access Processing System that manages L3 computing in and interfaces to the Data Access Center, specifying automated operations sequences, how human intervention can occur, and processes to handle changes and updates. (Story points are for KTL drafting and initial contributions)",3 DM-5493,"Develop functional breakdown for Observation Processing System","Write sections that can be incorporated into LDM-148 describing the functional breakdown of the Observation Processing System, including, for each major element: * overall function * inputs, outputs, and control interfaces * components used * descriptions of functions to be performed (Story points are for KTL drafting and initial contributions)",3 DM-5494,"Develop functional breakdown for Batch Processing System","Write sections that can be incorporated into LDM-148 describing the functional breakdown of the Batch Processing System, including, for each major element: * overall function * inputs, outputs, and control interfaces * components used * descriptions of functions to be performed (Story points are for KTL drafting and initial contributions)",2 DM-5495,"Develop functional breakdown for Data Backbone","Write sections that can be incorporated into LDM-148 describing the functional breakdown of the Data Backbone, including, for each major element: * overall function * inputs, outputs, and control interfaces * components used * descriptions of functions to be performed (Story points are for KTL drafting and initial contributions)",2 DM-5496,"Develop functional breakdown for Data Access Center Processing System","Write sections that can be incorporated into LDM-148 describing the functional breakdown of the Data Access Center Processing System, including, for each major element: * overall function * inputs, outputs, and control interfaces * components used * descriptions of functions to be performed (Story points are for KTL drafting and initial contributions)",2 DM-5497,"Develop DPS-WG documents","Create documents needed to accomplish the goals of the DPS-WG.",24 DM-5498,"Coordinate completion of operations concepts","Coordinate the creation of a new version of LDM-230 incorporating DPS-WG-generated operations concepts.",2 DM-5499,"Coordinate completion of functional breakdowns","Coordinate the creation of a new version of LDM-148 incorporating DPS-WG-generated functional breakdowns.",2 DM-5501,"Solve the metadata sanitization problem","Applications need access to visit specific metadata: e.g. pointing, airmass, exposure length. This information is typically carried around in a FITS header, but there are no conventions on spelling or even necessarily units of these metadata key, value pairs. There needs to be a easy to use metadata sanitization process that allows data from many different systems to present a standardized interface to observation metadata to the algorithm code.",100 DM-5502,"Collect usage of header metadata","Collect a comprehensive set of exposure oriented metadata used by science code. This should also include metadata that is not currently needed but that could be utilized in the future. In practice, I suspect this will involve looking for all calls to PropertySet.get since that is how FITS header metadata is currently passed around.",5 DM-5503,"Implement single interface to sanitized exposure metadata","Currently metadata associated with exposures is accessed in a few different ways: through the Calib object, through the Wcs object, and through the metadata object. In conjunction with DM-5502, which is to figure out what metadata is needed, this will provide a single interface to exposure oriented metadata. One tricky thing is that the Calib object has some subset of the metadata we'll need in a sanitized form, but we won't want to have to remember where to look for metadata. If we can't extend the Calib object to hold all sanitized metadata, we should create a new metadata object to store all sanitized metadata and remove the pieces from Calib that are currently there.",10 DM-5505,"Verification via precursor datasets","This epic covers covers timeboxed investigative activities into processing of precursor datasets with the LSST stack. ",26 DM-5506,"DAX & DB Docs (ABH)","* Add memman documentation (in LDM-135) * Refresh XRDSSI documentation (in LDM-135)",6 DM-5507,"Design Discussions (AndyS, March)",NULL,3 DM-5508,"QA Tasks & Supertasks","This epic covers stack-side work for the squash MVP (DM-5555) (JS:8, MWV:14)",22 DM-5509,"alert production database next steps (April)","Place-holder for additional alert production database work after investigate design task completes. We should split this into smaller stories for a total of 18 points this cycle.",7 DM-5510,"QA Tasks & Supertasks II","At this time this is epic is a bucket to keep track of backlog for validate_drp etc. ",22 DM-5511,"Design discussions (Brian, March)",NULL,1 DM-5512,"Design Discussions (John, March)",NULL,3 DM-5514,"Validate shared scan implementation on IN2P3 cluster",NULL,9 DM-5515,"prepare Slack RFC"," https://jira.lsstcorp.org/browse/RFC-140",1 DM-5516,"Design Discussions (Fritz, March)",NULL,3 DM-5517,"Design Discussions (Nate, March)",NULL,3 DM-5518,"Create proposal & RFC for Butler API to define output dataset type","this story represents a spike to 1. ad-hoc gathering of requirements to create butler API that allows a task to define an output dataset type 2. do any proof-of-concept mock up needed 3. write an RFC & gather input then A. If there is major dissent, create another design spike story or B. Close the RFC, and green light work on DM-4180",11 DM-5519,"SQuaRE documentation & design documents","This epic involves planning, design and usage documentation on SQuaRE products and services. (JMP:4,FE:15,JS:6)",25 DM-5520,"SQuaSH design proposal document"," Document capturing situation as of beginning of X16 can be found at: https://dmtn-016.lsst.io Further extension is planned in F16 to cover X16 development as well as consequences of the LDM-151 rewrite. ",10 DM-5521,"Python wrappers for sphgeom","This issue is a pre-req of DM-3472",15 DM-5523,"Weekly and monthly releases"," Some manual process at the rate of 1 SP / month is still involved in the releases until the automating publishing process is complete. ",6 DM-5526,"Add documentation to BinnedWcs","DM-5282 ported functionality from HSC to work in ""super-pixels"" which are the result of binning in wcs. This functionality was introduced in binnedWcs.(cc/h). This functionality needs proper doxygen documentation added.",1 DM-5530,"Documentation of Firefly functions and API (F16)","We are concentrating on the coding in X16. This epic will be capture the effort to write the document for using Firefly functions and API. ",40 DM-5532,"Change star selectors to return stars instead of PSF candidates","Implement RFC-154: - Make star selectors tasks, but continue to use and prefer a registry - Add an abstract base class for star selectors with the following methods: - {{selectStars}} abstract method that takes a catalog of sources and returns a {{lsst.pipe.base.Struct}} containing a catalog of stars - {{run}} concrete method that takes a catalog of sources and an optional name of a flag field, calls {{selectStars}} to select stars, then sets the flag field (if given) for stars - {{makePsfCandidates}} make a list of psf candidates from a catalog of stars (does no selection, other than skipping stars that cannot be made into candidates, and logging the rejects) ",4 DM-5533,"Add HTM indexing to sphgeom","To include Python wrappers, in support of DM-5052",10 DM-5534,"Design Discussions (Fritz, April)",NULL,6 DM-5535,"Design Discussions (Fritz, May)",NULL,6 DM-5536,"DAX & DB Docs (Fritz, April)",NULL,8 DM-5537,"DAX & DB Docs (Fritz, May)",NULL,8 DM-5538,"Finish data distribution prototype (April)",NULL,6 DM-5539,"Finish data distribution prototype (May)",NULL,6 DM-5540,"Design Discussions (John, April)",NULL,6 DM-5541,"Design Discussions (John, May)",NULL,6 DM-5542,"AFW rgb.py has undefined variable that breaks a test in some situations","The {{rgb.py}} test is failing for me with current AFW master: {code} tests/rgb.py .E............ ====================================================================== ERROR: testMakeRGBResize (__main__.RgbTestCase) Test the function that does it all, including rescaling ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/rgb.py"", line 313, in testMakeRGBResize with Tempfile(fileName, remove=True): NameError: global name 'Tempfile' is not defined ---------------------------------------------------------------------- Ran 16 tests in 7.296s FAILED (errors=1) {code} {{Tempfile}} is definitely only used in line 313. It was introduced with commit c9864f49. I'm not entirely sure how this is not picked up by Jenkins as the test will run if matplotlib and scipy are installed and Jenkins does have those. ",1 DM-5543,"Design Discussions (AndyS, April)",NULL,6 DM-5544,"Design Discussions (AndyS, May)",NULL,6 DM-5545,"Alert production database next steps (May)",NULL,7 DM-5546,"Design Discussions (Brian, April)",NULL,2 DM-5547,"Design Discussions (Brian, May)",NULL,2 DM-5548,"persistence improvements to butler config system","requirements: - easy to know what to provide - fails fast - has a way to be backward compatible with persisted configs & existing scripts existing issues to fix: - currently config creation is verbose, difficult to read, and difficult to format properly - has the butler class hierarchy baked into the format",10 DM-5550,"Design Discussions (Nate, April)",NULL,6 DM-5551,"Design Discussions (Nate, May)",NULL,6 DM-5552,"Add renderer option to js table","TablePanel and BasicTable now accept optional renderers. For each column, you can set a custom renderer for the header, cell, or both. Also, created several commonly used renderer for images, links, and input field.",2 DM-5553,"Z-scale stretch for image display","The z-scale stretch in current system is different from the one in OPS",8 DM-5554,"Assist in document investigation of logging, monitoring and metrics technologies and architecture","Assist with tech note SQR-007 and document investigation of logging, monitoring and metrics technologies and architecture.",2 DM-5555,"SQuaSH MVP","This is an epic that covers setting up a minimally viable QA environment for executing processing, calculating metrics, storing them and displaying them. This serves a dual purpose: - It results in a limited but still useful production service that developers can take advantage of - It allows us to assess our initial technology stack for suitability for further development. The test case was picked to be a supertask version of the tests described in DM-4730 (informally ci_lauren) used during the merging of the HSC fork. This was meant to also allow us to use and give feedback on the supertask architecture. When it became obvious that we would not take delivery of that infrastructure in time for X16 work, we switched the test case to one of the KPMs calculated in validate_drp. This switch does not affect the engineering aims of this prototype. The MVP based on validate_drp can eventually be extended to service other types of KPM measurement for regression testing and release characterisation. (JH:32, JMP:16, JS:32, MVW:14, AF:20) Outcome: MVP stood up on squash.lsst.codes. Currently collecting AM1, AM2 and PA1 KPMs using validate_drp on validation_data_cfht. After a short period of evaluation of the performance of the toolchains in production we will proceed with more data, more metrics and more features for F16. ",100 DM-5556,"SQuaRE ad-hoc developer requests","This is a bucket epic for requests from developers/ science users that come up mid-cycle and cannot wait until the new cycle, security vulnerabilities, critical bug fixes, etc. (JH:16,JMP:8,FE:3)",27 DM-5559,"Present Supertask design to DMLT","Present the Supertask design to the November 2015 DMLT in-person meeting. Covers preparation of a presentation and related discussions preceding and immediately following the meeting.",6 DM-5560,"Participate in October 2015 OCS-subsystems teleconference","Prepare for, attend, and follow up on the OCS-subsystems teleconference on October 8, 2015.",2 DM-5561,"Write tech note on modifications required to use py.test framework","Following the investigatory work into switching our Python test files to be compliant with pytest, whilst still using {{unittest}}, a tech note needs to be written explaining the required changes.",10 DM-5562,"Participate in November 2015 OCS-subsystems teleconference (LSE-70, LSE-209)","Prepare for, attend, and follow up on the OCS-subsystems teleconference on November 11, 2015. This story covers work related to LSE-70 and LSE-209; LSE-74 work was also done under a separate epic.",4 DM-5563,"Participate in November 2015 OCS-subsystems teleconference (LSE-74)","Prepare for, attend, and follow up on the OCS-subsystems teleconference on November 11, 2015. This story covers work related to LSE-74; LSE-70 and LSE-209 work was also done under a separate epic.",1 DM-5564,"Participate in December 2015 OCS-subsystems teleconference (LSE-70, LSE-209)","Prepare for, attend, and follow up on the OCS-subsystems teleconference on December 9, 2015. This story covers work related to LSE-70 and LSE-209; LSE-74 work was also done under a separate epic.",2 DM-5565,"Participate in December 2015 OCS-subsystems teleconference (LSE-74)","Prepare for, attend, and follow up on the OCS-subsystems teleconference on December 9, 2015. This story covers work related to LSE-74; LSE-70 and LSE-209 work was also done under a separate epic.",1 DM-5566,"Review of LSE-70 and LSE-209 drafts, September 2015","Arrange, prepare for, and attend a joint call with the Camera team to review the end-of-summer-2015 drafts of LSE-70 and LSE-209 from the OCS group.",3 DM-5567,"CCB review of LCR-567 (LSE-70) and LCR-568 (LSE-209)","Review the LSE-70 and LSE-209 drafts submitted with change requests LCR-567 and LCR-568 in January 2016.",2 DM-5568,"CCB review of LCR-603 (LSE-74)","Review LCR-603, ""LSE-74 document revision""",2 DM-5569,"LSE-70, LSE-209 refinements X16","There are open LCRs for cleanups to the versions of LSE-70 and LSE-209 approved by the CCB in February 2016. An initial teleconference will be held on 30 March 2016 with the OCS group to discuss these.",4 DM-5578,"making PSF candidates should be simpler","The code to make PSF candidates is too complicated and repeated in too many places (even after DM-5532). Every time lsst.meas.algorithms.makePsfCandidate is called (except in a few tests) it is called as follows: {code} cand = measAlg.makePsfCandidate(source, mi) if cand.getWidth() == 0: cand.setBorderWidth(borderWidth) cand.setWidth(kernelSize + 2*borderWidth) cand.setHeight(kernelSize + 2*borderWidth) im = cand.getMaskedImage().getImage() max = afwMath.makeStatistics(im, afwMath.MAX).getValue() if not numpy.isfinite(max): continue {code} This should to be centralized somewhere. I suggest adding this code to {{meas.algorithms.makePsfCandidate}} itself (which could delegate some work to a private function, if desired).",2 DM-5579,"Add Error and Working feedback to FITS visualizer","* Add working message when plot is loading, (downloading..., plotting..., etc) * Add error message when plot fails * for multi-viewer remove and failed plot cells * work out if image select panel should become visible again. * Any thing else the is plotting feedback related",6 DM-5580,"Docgen draft from EA content for LSE-140","Create a docgen from the LSE-140 content in Enterprise Architect.",2 DM-5581,"SQuaRE Communication and Publication Platforms Document and Presentation","[SQR-011|http://sqr-011.lsst.io] documents the various communication and publishing platforms that SQuaRE operates on behalf of DM. This ticket will complete v1 of the document (DM-4721 created a time-boxed first draft) and also include work to present the document to LSST management.",4 DM-5582,"Support LCR-385","Support getting LCR-385 against LSE-78 through the CCB.",3 DM-5584,"Create a reusable upload file component","This component will upload and validate the file as part of the input's validation process. It will return a token generated by the server which will resolve to the uploaded file if the upload success. ",4 DM-5585,"SQuaRE Communication and Publication Platforms Document and Presentation - Clone","This is a clone of DM-5581 tracking [~frossie]'s SPs",5 DM-5586,"Fix obs_decam butler level","There is a bug in {{obs_decam/policy/DecamMapper.paf}}, causing some butler features for the ""visit"" level or above working incorrectly. The {{hdu}} key is irrelevant for the visit level or above, but wasn't included in the policy file. Because of this bug, the {{DemoTask}} in {{ctrl_pool}} (ctrlPoolDemo.py) runs incorrectly with DECam data. It incorrectly treats dataRef with different {{hdu}}s as they are from different visits, hence reads each ccd image multiple times (61 times for one visit with 61 hdu). Instead, each ccd image should be read once. Besides fixing the policy file, I also added an optional test that only runs if {{testdata_decam}} is set up. The part with level=""visit"" in the test fails without the ticket changes in the policy. (p.s. The raw data file in {{testdata_decam}} is modified and has only 2 hdus.) ",3 DM-5588,"Add lmfit package to the stack","The current implementation of the new {{DipoleFitTask}} for {{ip_diffim}} uses {{lmfit}} to perform parameter estimation (least-squares minimization). {{lmfit}} is essentially an API on top of {{scipy}}'s optimizer, adding functionality such as parameter boxing (constraints) and improved estimates of parameter uncertainties. It would be nice to include this small, pure-python package in the stack rather than investigating and re-implementing the optimization using {{scipy}} or {{minuit2}} (which are the two optimizers that I know of that are in the stack already).",4 DM-5590,"Fix afw build issues with recent clang","{{afw}} fails to build with recent versions of clang: {code} include/lsst/afw/image/MaskedImage.h:553:65: error: '_loc' is a protected member of 'lsst::afw::image::MaskedImage<unsigned short, unsigned short,       float>::MaskedImageLocatorBase<boost::gil::memory_based_2d_locator<boost::gil::memory_based_step_iterator<boost::gil::pixel<unsigned short,       boost::gil::layout<boost::mpl::vector1<boost::gil::gray_color_t>, boost::mpl::range_c<int, 0, 1> > > *> >,       boost::gil::memory_based_2d_locator<boost::gil::memory_based_step_iterator<boost::gil::pixel<unsigned short,       boost::gil::layout<boost::mpl::vector1<boost::gil::gray_color_t>, boost::mpl::range_c<int, 0, 1> > > *> >,       boost::gil::memory_based_2d_locator<boost::gil::memory_based_step_iterator<boost::gil::pixel<float,       boost::gil::layout<boost::mpl::vector1<boost::gil::gray_color_t>, boost::mpl::range_c<int, 0, 1> > > *> >, Reference>'                                      const_VarianceLocator(iter._loc.template get<2>()) {code} and issues with statistics.i so far, more errors may turn up as these are cleared. These problems are apparent with {{Apple LLVM version 7.3.0 (clang-703.0.29)}} (as shipped with the latest release of XCode, hence this now becoming an issue) and {{clang version 3.8.0 (branches/release_38 262722)}} (a recent release from LLVM; note that Apple uses its own versioning scheme). {{clang version 3.7.1 (tags/RELEASE_371/final)}} is not affected.",1 DM-5591,"Archive in a box v1 (F16)","Several times we were asked a question about Firefly: Great software. How could I use it for my data now? We want to create a recipe and stubs of code (archive in a box) so others can take it, with minimal configuration changes and minimal customized data access code, to have a simple archive UI for their data. It will come with all the built-in images and catalogs access, all the image, catalog, and XY plot functions. ",40 DM-5593,"fix issue where butler repository search returns list for single item","Backwards compatible behavior is that when butler returns a single item, it is NOT in a list. A recent change (when the Repository class was added) broke this behavior. Change it back so that if an operation in repository would return a list with a single item, it pulls it from the list. Note this is only related to the case where a repository's parentJoin field is set to 'outer' and since no one is using this yet (they should not be, anyway) then the point is moot. ",1 DM-5594,"Fix qserv service timeout issue","After Qserv services have been running over ~couple of days, new queries fail and can also lead to a crash. Investigate and implement a solution.",5 DM-5595,"daf_persistence build failure on OSX","I see the following build failure in {{daf_persistence}} on OSX 10.11: {code} c++ -o python/lsst/daf/persistence/_persistenceLib.so -bundle -F/ -undefined suppress -flat_namespace -headerpad_max_install_names python/lsst/daf/persistence/persistenceLib_wrap.os -Llib -L/private/tmp/ssd/swinbank/shared_stack/DarwinX86/mariadbclient/10.1.11-2-gd04d8b7/lib -L/private/tmp/ssd/swinbank/shared_stack/DarwinX86/pex_policy/2016_01.0+4/lib -L/private/tmp/ssd/swinbank/shared_stack/DarwinX86/pex_logging/2016_01.0+4/lib -L/private/tmp/ssd/swinbank/shared_stack/DarwinX86/daf_base/2016_01.0+4/lib -L/private/tmp/ssd/swinbank/shared_stack/DarwinX86/utils/2016_01.0+4/lib -L/private/tmp/ssd/swinbank/shared_stack/DarwinX86/pex_exceptions/2016_01.0+3/lib -L/private/tmp/ssd/swinbank/shared_stack/DarwinX86/base/2016_01.0+3/lib -L/private/tmp/ssd/swinbank/shared_stack/DarwinX86/boost/1.59.lsst5/lib -L/tmp/ssd/swinbank/shared_stack/DarwinX86/miniconda2/3.19.0.lsst4/lib/python2.7/config -ldaf_persistence -lboost_serialization -lmysqlclient_r -lpex_policy -lpex_logging -lboost_filesystem -lboost_system -ldaf_base -lutils -lpex_exceptions -lbase -lboost_regex -lpthread -ldl -lpython2.7 ld: file not found: libz.1.dylib for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) scons: *** [python/lsst/daf/persistence/_persistenceLib.so] Error 1 scons: building terminated because of errors. {code} This happens with the current master ({{3484020}} at time of writing), but also with a recent weekly ({{3878625}}). ",1 DM-5604,"Remove obsolete install scripts from ~/src/qserv/admin/tools/","Internet-free install scripts are unused and should be removed with related documentation.",1 DM-5605,"runQueries.py fails on IN2P3 cluster","Launching runQueries.py produces some errors: {code:bash} fjammes@ccosvms0070:~/src/qserv/admin/tools/docker/deployment/in2p3 (tickets/DM-5402 *=)$ ./run-test-queries.sh +--------------------+--------------------+ | ra | decl | +--------------------+--------------------+ | 29.308806347275485 | -86.30884046118973 | +--------------------+--------------------+ real 1m20.725s user 0m0.004s sys 0m0.012s Output directory: /afs/in2p3.fr/home/f/fjammes/runQueries_out Exception in thread Thread-12: Traceback (most recent call last): File ""/usr/lib64/python2.7/threading.py"", line 811, in __bootstrap_inner self.run() File ""/usr/lib64/python2.7/threading.py"", line 764, in run self.__target(*self.__args, **self.__kwargs) File ""/afs/in2p3.fr/home/f/fjammes/src/qserv/admin/tools/docker/deployment/in2p3/runQueries.py"", line 185, in runQueries db='LSST') File ""/qserv/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/__init__.py"", line 81, in Connect return Connection(*args, **kwargs) File ""/qserv/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/connections.py"", line 187, in __init__ super(Connection, self).__init__(*args, **kwargs2) OperationalError: (2013, ""Lost connection to MySQL server at 'reading authorization packet', system error: 0"") Exception in thread Thread-23: Traceback (most recent call last): File ""/usr/lib64/python2.7/threading.py"", line 811, in __bootstrap_inner self.run() File ""/usr/lib64/python2.7/threading.py"", line 764, in run self.__target(*self.__args, **self.__kwargs) File ""/afs/in2p3.fr/home/f/fjammes/src/qserv/admin/tools/docker/deployment/in2p3/runQueries.py"", line 185, in runQueries db='LSST') File ""/qserv/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/__init__.py"", line 81, in Connect return Connection(*args, **kwargs) File ""/qserv/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/connections.py"", line 187, in __init__ super(Connection, self).__init__(*args, **kwargs2) OperationalError: (1105, '(proxy) all backends are down') Exception in thread Thread-16: Traceback (most recent call last): File ""/usr/lib64/python2.7/threading.py"", line 811, in __bootstrap_inner self.run() File ""/usr/lib64/python2.7/threading.py"", line 764, in run self.__target(*self.__args, **self.__kwargs) File ""/afs/in2p3.fr/home/f/fjammes/src/qserv/admin/tools/docker/deployment/in2p3/runQueries.py"", line 185, in runQueries db='LSST') File ""/qserv/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/__init__.py"", line 81, in Connect return Connection(*args, **kwargs) File ""/qserv/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/connections.py"", line 187, in __init__ super(Connection, self).__init__(*args, **kwargs2) OperationalError: (2013, ""Lost connection to MySQL server at 'reading authorization packet', system error: 0"") Exception in thread Thread-21: Traceback (most recent call last): File ""/usr/lib64/python2.7/threading.py"", line 811, in __bootstrap_inner self.run() File ""/usr/lib64/python2.7/threading.py"", line 764, in run self.__target(*self.__args, **self.__kwargs) File ""/afs/in2p3.fr/home/f/fjammes/src/qserv/admin/tools/docker/deployment/in2p3/runQueries.py"", line 185, in runQueries db='LSST') File ""/qserv/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/__init__.py"", line 81, in Connect return Connection(*args, **kwargs) File ""/qserv/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/connections.py"", line 187, in __init__ super(Connection, self).__init__(*args, **kwargs2) OperationalError: (1105, '(proxy) all backends are down') Exception in thread Thread-17: Traceback (most recent call last): File ""/usr/lib64/python2.7/threading.py"", line 811, in __bootstrap_inner self.run() File ""/usr/lib64/python2.7/threading.py"", line 764, in run self.__target(*self.__args, **self.__kwargs) File ""/afs/in2p3.fr/home/f/fjammes/src/qserv/admin/tools/docker/deployment/in2p3/runQueries.py"", line 185, in runQueries db='LSST') File ""/qserv/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/__init__.py"", line 81, in Connect return Connection(*args, **kwargs) File ""/qserv/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/connections.py"", line 187, in __init__ super(Connection, self).__init__(*args, **kwargs2) OperationalError: (2013, ""Lost connection to MySQL server at 'reading authorization packet', system error: 0"") Exception in thread Thread-19: Traceback (most recent call last): File ""/usr/lib64/python2.7/threading.py"", line 811, in __bootstrap_inner self.run() File ""/usr/lib64/python2.7/threading.py"", line 764, in run self.__target(*self.__args, **self.__kwargs) File ""/afs/in2p3.fr/home/f/fjammes/src/qserv/admin/tools/docker/deployment/in2p3/runQueries.py"", line 185, in runQueries db='LSST') File ""/qserv/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/__init__.py"", line 81, in Connect return Connection(*args, **kwargs) File ""/qserv/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/connections.py"", line 187, in __init__ super(Connection, self).__init__(*args, **kwargs2) OperationalError: (2013, ""Lost connection to MySQL server at 'reading authorization packet', system error: 0"") Exception in thread Thread-18: Traceback (most recent call last): File ""/usr/lib64/python2.7/threading.py"", line 811, in __bootstrap_inner self.run() File ""/usr/lib64/python2.7/threading.py"", line 764, in run self.__target(*self.__args, **self.__kwargs) File ""/afs/in2p3.fr/home/f/fjammes/src/qserv/admin/tools/docker/deployment/in2p3/runQueries.py"", line 185, in runQueries db='LSST') File ""/qserv/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/__init__.py"", line 81, in Connect return Connection(*args, **kwargs) File ""/qserv/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/connections.py"", line 187, in __init__ super(Connection, self).__init__(*args, **kwargs2) OperationalError: (2013, ""Lost connection to MySQL server at 'reading authorization packet', system error: 0"") {code}",4 DM-5607,"check & correct comparison operators in daf_persistence and daf_butlerUtils","per comments in DM-5593, an incorrect comparison operator was found, that used {{is}} instead of {{==}} in a string comparison (e.g. {{var is 'left'}} which is incorrect, it should be {{var == 'left'}}. This needs to be corrected in {{Repository}} (see DM-5593 for details), and the rest of daf_persistence and daf_butlerUtils should be checked for correct use of is vs. ==.",1 DM-5608,"plan and RFC for ""data repository based on version""","create way using repo of repos to get only the posix root in the returned cfg, update example/test code. write an RFC, review with KT post RFC & gather feedback. incorporate feedback and/or create another design story if needed.",12 DM-5609,"Investigate clang issues regarding friendship and protected members ","In DM-5590, we worked around a problem in which clang 3.8 refused to access protected members of a cousin class given a friend declaration in the base. To our best understanding at time of writing, the code is valid: it seems possible that this is a bug in clang. Investigate what went wrong, produce a minimal test case, and (if appropriate) report this as an upstream bug.",3 DM-5610,"x16 Operations Planning in LOPT, TOWG, and DM replanning (ConOps development)","Develop ConOps for DM system, including Bulk Batch System, Data Backbone, L3 Hosting, etc. Develop use cases for TOWG. Continued planning for operations, focusing on Data Processing and Products directorate. Don Petravick, Margaret Gelman, Hsin-Fang Chiang, Stephen Pietrowicz, Jaggi Yedetore, Paul Wefel ",54 DM-5611,"x16 Joint Coordination Council","Coordination with CC-IN2P3. Don Petravick, Jason Alt ",7 DM-5612,"Design specification and requirements analysis of Bulk Batch System","Functional breakdown of Bulk Batch System, including L2 processing, calibration processing, etc. Detailed design, plan, and schedule. Don Petravick, Margaret Gelman, Jason Alt, Hsin-Fang Chiang, Stephen Pietrowicz, Rob Kooper, Paul Wefel",68 DM-5613,"Design specification and requirements analysis of Data Backbone","Functional breakdown of Data Backbone. Detailed design, plan, and schedule. Jason Alt, Don Petravick, Margaret Gelman, Paul Wefel",23 DM-5614,"x16 middlware/workflow package definition and development","Defining future middleware package to support science pipeline processing. Maintaining and adding to current middleware packages. Prototyping processing sequences with DECam data. ",100 DM-5616,"x16 LSST Identity and Access Management Program development",NULL,8 DM-5617,"Further requirements analysis of the L1 System","Additional design specification of parts of the L1 system that we haven't looked at in detail yet, such as EFD replication, Observatory Operations Server, Auxiliary Telescope processing, telemetry processing, and Commissioning support. Margaret Gelman, Stephen Pietrowicz, James Parsons, Paul Wefel ",20 DM-5618,"WAN emulation testing, project 1","Project 1 of WAN emulation. See https://confluence.lsstcorp.org/display/JP/WAN+Emulator+test+plan+-+January+2016 for details. James Parsons, Paul Wefel ",21 DM-5620,"L1 Basic Message Topology (x16)","Includes system status and message dictionary, main programs for all L1 entities, and message interaction between L1 entities.",95 DM-5622,"Network and Service Monitoring (Comfort Console)","This is the development and integration work required for architecting and building a Network monitoring center.",36 DM-5623,"Procure FY16 capabilities","Quotes, discussions with vendors, details specs of procurements. ",42 DM-5624,"Deploy FY16 Cluster Services",NULL,28 DM-5625,"Migrate to distributed filesystem",NULL,20 DM-5626,"Deploy FY16 Verification Cluster",NULL,36 DM-5627,"Deploy FY16 Object Store Discovery Infrastructure","Deploy infrastructure for FY16 object store evaluation. Deliverable: Object Store discovery infrastructure Staff: 3 NCSA ICI engineers (networking, storage, systems) Effort: 20 days Planned Start: 6/1/2016 Planned End: 6/30/2016",40 DM-5628,"x16 ISO Work",NULL,20 DM-5632,"test run coaddDriver and multiBandDriver with DECam data","Preparation work to learn about ctrl_pool and pipe_drivers packages. - Install ctrl_pool and pipe_drivers packages on my OSX desktop (no slurm). - Run the ctrl_pool mpiexec example to verify if the mpi is working. - Obtain a HSC data repo from ci_hsc as sanity checks. - Construct a small DECam data repo, using raw Stripe82 data consisting of two visits, one band, one patch. - Run ctrlPoolDemo.py with the HSC data repo and then the DECam data repo. - Run the pipe_drivers scripts {{coaddDriver}} and {{multiBandDriver}} with the HSC data repo and then the DECam data repo. - All with the default batch system SMP to run on a single machine.",12 DM-5633,"Add data products and config in obs_decam for multi-band processing","Add necessary data products and default config in order to run forcedPhotCcd, coaddDriverTask, and multiBandDriverTask with DECam data. ",3 DM-5636,"Python version checking in newinstall.sh is not quite right","There is a recent report on community where {{newinstall.sh}} reports that the python version is too old despite the user having a modern Anaconda python in their path. In commit e6fc9ed2 the code was changed to check {{$PYTHON}} for version compatibility but that is not correct as the python that will be used for the actual build is the python in their path. {{$PYTHON}} is defined purely as the python to use for EUPS installation. In the reported error {{$PYTHON}} was not set and their {{/usr/bin/python}} was too old. Confusingly the error message reporting the version problem actually reported the version information for the python in the path and not the {{$PYTHON}} python. The simple fix is to revert e6fc9ed2. I already made significant comments on this topic in the original https://github.com/lsst/lsst/pull/19 but I really do have to insist on either reverting that PR or at least fixing the error messages to use a consistent python (I'd argue that this is still wrong but at least consistent). The current situation is at best confusing and at worst pointless and wrong. The version test only makes sense if we are testing that the default python in the path is the correct version to build the stack. {{$PYTHON}} was originally designed to allow a different python to be used to build EUPS. Even that is no longer an issue as EUPS can work with Python >= 2.6 now.",1 DM-5638,"Make file upload show feedback when file is uploading",NULL,2 DM-5640,"Build a tool to automatically run autopep8 on LSST Stack","Develop a lsst-autopep8 command in [sqre-codekit|https://github.com/lsst-sqre/sqre-codekit] that can run [autopep8|https://github.com/hhatto/autopep8] in an automated fashion across all of the LSST Stack repositories according to the PEP 8 exceptions determined in RFC-162.",1 DM-5641,"finish up afw.table to astropy.table view support","At an LSST/AstroPy summit hack session, we've put together a functional system for viewing afw.table objects as astropy.table objects on branch u/jbosch/astropy-tables of afw and https://github.com/astropy/astropy/pull/4740. Before merging, we should add support for ""object"" columns for subclasses to hold e.g. Footprints in SourceCatalog, and add some documentation. We may also want to add a convenience method to return an astropy.table.Table directly.",1 DM-5642,"use AstroPy-compliant strings for units in afw.table","With DM-5641, we'll soon be able to get astropy.table views into afw.table objects. That will be a bit more useful if astropy can understand the unit strings we give it, and since we currently don't use those strings as anything more than textual information for humans, we might as well standardize on the terms they've already selected.",4 DM-5643,"add method to convert Property[Set,List] to nested dict","In interfacing with AstroPy it'd be useful to easily convert PropertySet and PropertyList to nested dict and OrderedDict (respectively), converting elements with multiple values to lists in the process.",2 DM-5645,"Add fine-grained authorization to ltd-keeper users","The initial MVP of ltd-keeper had all-or-nothing authentication; any user was effectively an admin user. It would be useful have fine grained roles that each API user could have (for example, one API user might be able to add a build, but not create an edition or product or add another user). The phases of this ticket at: 1. Design a set of roles that cover current functionality 2. Add these roles to the User DB model and user creation API 3. Authorize users against these roles in specific API calls",2 DM-5647,"Research & Design for object storage & transport factorization","Start research and design proposals & prototyping on the butler back end factorization problem; need to be able to configure butler to put and get different object types, different storage formats, and different storage locations. Do intermediate KT reviews. Initial design and stubbed implementation to include the following: *ExposureF* to/from *Fits* file on *local filesystem* (Posix) *ExposureF* to/from *Memory* *SourceCatalog* to/from *database* *SourceCatalog* to/from *Fits* on *local filesystem* (Posix) *SourceCatalog* to/from *Memory*",9 DM-5648,"Finish stubs and write role description for butler back end factorization",NULL,8 DM-5649,"visit AP team and work on processing DECam data","March 13-17, 2016. Work on various topics about processing DECam data: - Improve documentations on processing raw DECam data, especially about the steps of ingesting calibration data - Identify future work on improving processing raw data. Updates about Instrumental Signature Removal tasks. - Learn how to run difference imaging pipeline with DECam data - Try jointcal (Simultaneous Astrometry meas_simastrom package from IN2P3) with DECam data and identity necessary code changes for doing jointcal with DECam data - Use the preliminary jointcal astrometry results to examine DECam data’s distortion - Also more general discussions on data processing",8 DM-5650,"SUIT vision document","Writing down SUIT vision that the group has discussed and shaped in last year off and on. SUIT will use it as guidance for system design.",4 DM-5651,"run jenkins for PRs on all EUPS products - part I",NULL,1 DM-5652,"Implement RFC-167","Implement RFC-167 for adding esutil to the stack. This will be done in the same way as proposed to add scipy.",2 DM-5655,"Reduce code duplication in StarSelectors","Both {{ObjectSizeStarSelector}} and {{SecondMomentStarSelector}} have logic to transform measured moments from pixel coordinates to TAN_PIXELS in order to remove optical distortion. That's generically useful for any star selector that works on measured moments, and we shouldn't have to repeat it everywhere it is used.",2 DM-5657,"Improve Large Test Scale query script","This script is currently located in: admin/tools/docker/deployment/in2p3/runQueries.py Here's some improvments: - use lsst/db instead of mysqlpython - externalize queries and other parameters in a config file - add an option to make script stop after a few queries (in order to have deterministic query results for Large Scale integration tests) - any other minor improvments...",8 DM-5658,"Update tables of packages that depend on scipy","Now that the {{scipy}} package has been added (DM-5446), the table files of other packages to be fixed as soon as possible so that we have an idea of what is silently depending on {{scipy}}. These include {{afw}}, {{ip_diffim}}, {{meas_modelfit}}, {{mops_daymops}}, {{pipe_tasks}}, {{shapelet}} and {{sims_photUtils}}. Many of these are setupOptional that we should consider making mandatory. Some will be setupRequired.",2 DM-5659,"multiple dialog are not working well together","When several dialogs are up together. The most recently click one should be one top. When table are in the dialogs such a fits header view. The scroll bars will go over other dialogs. This needs some though and work. Another thing- when a message dialog is show because of a dialog error. It should center on the dialog. Update- I don't think I will do the error centering now. I am going to leave that and see if it is a real problem.",3 DM-5660,"Add motivated model fits to validate_drp photometric and astrometric scatter/repeatability analysis and plots","Implement well-motivated theoretical fits to the astrometric and photometric performance measurements based on derivations from LSST Overview paper. http://arxiv.org/pdf/0805.2366v4.pdf Photometric errors described by Eq. 5 sigma_rand^2 = (0.039 - gamma) * x + gamma * x^2 [mag^2] where x = 10^(0.4*(m-m_5)) Eq. 4 sigma_1^2 = sigma_sys^2 + sigma_rand^2 Astrometric Errors error = C * theta / SNR Based on helpful comments from [~zivezic] {quote} I think eq. 5 from the overview paper (with gamma = 0.039 and m5 = 24.35; the former I assumed and the latter I got from the value of your analytic fit that gives err=0.2 mag) would be a much better fit than the adopted function for mag < 21 (and it is derived from first principles). Actually, if you fit for the systematic term (eq. 4) and gamma and m5, it would be a nice check whether there is any “weird” behavior in analyzed data (and you get the limiting depth, m5, even if you don’t go all the way to the faint end). Similarly, for the astrometric random errors, we’d expect error = C * theta / SNR, where theta is the seeing (or a fit parameter), SNR is the photometric SNR (i.e. 1/err in mag), and C ~ 1 (empirically, and 0.6 for the idealized maximum likelihood solution and gaussian seeing). {quote}",5 DM-5663,"Config override fixes needed due to new star selector","As of DM-5532 a few config files need updating to not refer to star selector config fields as registries (not ones run by our normal CI, which is how I missed this).",2 DM-5665,"Organize HSC docs ""hackathon""","Liase with SQuaRE to determine the most effective way to transfer HSC docs to LSST. Organize a hackathon session for DRP developers at which we get this done. Bring doughnuts.",1 DM-5666,"Take part in HSC docs hackathon","Participate in HSC docs transfer hackathon.",2 DM-5667,"Take part in HSC docs hackathon","Participate in HSC docs transfer hackathon. ",2 DM-5668,"Take part in HSC docs hackathon","Participate in HSC docs transfer hackathon.",2 DM-5669,"Take part in HSC docs hacakthon","Participate in HSC docs transfer hackathon.",2 DM-5670,"Take part in HSC docs hackathon","Participate in HSC docs transfer hackathon.",2 DM-5671,"Take part in HSC docs hackathon","Participate in HSC docs transfer hackathon. ",2 DM-5672,"Take part in HSC docs hackathon","Participate in HSC docs transfer hackathon. ",2 DM-5673,"Take part in HSC docs hackathon","Participate in HSC docs transfer hackathon.",2 DM-5674,"Prepare detailed L2 plan","At the scipi-wg meeting of 23 & 24 March 2017, [~jbosch] presented an overview of his plans for L2 processing. The next step is to refine those plans and prepare a more detailed ""deep-dive"" discussion of the L2 plans.",10 DM-5675,"Cannot enable shapeHSM because RegistryField fails validation","When running ci_hsc after setting-up the meas_extensions_shapeHSM, meas_extensions_photometryKron and dependencies using setup -v -r . in the respective cloned folders, I get {code} Cannot enable shapeHSM (RegistryField 'calibrate.detectAndMeasure.measurement.plugins' failed validation: Unknown key 'ext_shapeHSM_HsmMoments' in Registry/ConfigChoiceField For more information read the Field definition at: File ""/home/vish/code/lsst/lsstsw/stack/Linux64/pex_config/2016_01.0+3/python/lsst/pex/config/registry.py"", line 179, in __init__ ConfigChoiceField.__init__(self, doc, types, default, optional, multi) And the Config definition at: File ""/home/vish/code/lsst/lsstsw/stack/Linux64/meas_base/2016_01.0-13-g779ee14/python/lsst/meas/base/sfm.py"", line 109, in <module> class SingleFrameMeasurementConfig(BaseMeasurementConfig): ): disabling HSM shape measurements {code} Find out why this is happening and find a fix ",1 DM-5676,"Wrap example C++ code with pybind11","Same as DM-5471 but using pybind11",10 DM-5677,"Wrap example code with cffi","As per DM-5471, but using cffi.",10 DM-5681,"Provide single-visit processing capability as required by HSC","In DM-3368, we provided a means of running multiple processCcd tasks across an exposure, but without performing global calibration etc as provided by HSC's ProcessExposureTask. Please augment this with whatever additional capability is required to enable HSC data release processing.",2 DM-5685,"processCcd.py is failing on some CFHT u band images","processCcd.py is failing on some u band CFHT data, as reported by [~boutigny] on c.l.o: https://community.lsst.org/t/testing-dm-4692-the-new-processccdtask/507/24 See that posting for sample data to reproduce the problem.",8 DM-5686,"Accommodate pixel padding when unpersisting reference catalog matches","The reference object loader in {{meas_algorithm}}'s *loadReferenceObjects.py* grows the bbox by the config parameter pixelMargin: doc = ""Padding to add to 4 all edges of the bounding box (pixels)"" . This is set to 50 by default but is not reflected by the radius parameter set in the metadata, so some matches may reside outside the circle searched within this radius. This increase needs to be reflected in the radius set in the metadata fed into {{joinMatchListWithCatalog()}}. ",2 DM-5688,"Table performance on Firefly","Table seems to perform poorly on Firefox. Firefox gets into the complete refresh state only when the table is visible with charts only, fits view only or fits view and chart it does not happen Helpful article: http://benchling.engineering/performance-engineering-with-react/ changelog: - added react performance tools, React.addons.Perf - fix some performance issues: - skip render of selection boxes when not needed. - skip rendering of xyplot options when not needed. - skip wasted render called for table cell and headers. - will do a more in depth investigation in another ticket. - refactor table code and it's state. - move all table related states into table_space. - create sub reducers for each data domain - rename and move functions to better describe what it's doing - added 'title' to table. - show mask while loading ",6 DM-5689,"Table needs to fire another action when data completely loaded","When the data for a table is completely loaded fire another action such as TABLE_NEW_LOADED_DONE. This way the xyplots and the image overlays know to go fetch the data. 4/22/2026 from the pull request: added new action TABLE_NEW_LOADED to table; fired when table is completely loaded. added table error handling. fix active table not updating after an active tab is removed.",2 DM-5691,"AP Emergent work -- F16","There is emergent work that comes up as a side effect of other work. This epic will capture that effort in F16.",19 DM-5692,"Connect CatSim to StarFast simulation tool","CatSim can provide a fully realistic simulated catalog, which StarFast could use as an input for simulations. This ticket includes writing the code to connect to the CatSim database and updating the internal catalog format in StarFast to be compatible with CatSim.",4 DM-5693,"Write StarFast interface to ProcessCCD","The simulated images generated by StarFast need to be able to be run through the LSST stack, to test and make use of the existing measurement, fitting, stacking, and image differencing capabilities. This includes writing or updating a simulations obs package, and determining and supplying the required metadata.",6 DM-5694,"Run StarFast simulated images through diffim","Determine the metadata and dependencies needed to fully process two images simulated with StarFast through diffim. ",2 DM-5695,"Implement simple 1D DCR correction on simulated data","Nate Lust wrote a simple DCR correction recipe that runs in 1D in an ipython notebook. This ticket is to re-write the notebook in python modules that can be run on StarFast simulated images prior to image differencing. For this ticket, the simulated images will be 2D, but DCR will be purely along the x or y pixel grid, allowing columns or rows of pixels to be treated separately in 1D.",8 DM-5696,"Add support for blank image","We need to add blank image support.",2 DM-5697,"Extend simple DCR correction to 2D","In DM-5695 a simple DCR correction was applied to simulated images in the case that the effect was purely along the pixel grid and could be reduced to 1D. This ticket extends that work to the general 2D case. Possible approaches include resampling the ""science"" image to match the ""template"", or including neighboring pixels and computing their covariance. Ideally, multiple approaches will be implemented and tested.",6 DM-5698,"Add astrometric errors to StarFast","One concern with the proposed DCR correction is that it might fail in the presence of source position errors. This ticket is to add the capability to simulate a variety of types of position errors, such as atmospheric turbulence or an inaccurate WCS, to test the DCR implementation.",4 DM-5699,"Run many sky simulations through DCR correction to find edge cases","Once a complete DCR correction prototype is finished, we will want to run many different sky simulations from StarFast with different densities of sources, noise properties, airmasses, and astrometric errors to find the limitations and edge cases where it fails. There are likely to be several thousand simulations needed which will take an as-yet undefined number of CPU hours, but this ticket is for the work in setting up and analyzing the results from the run.",4 DM-5700,"Put ImageSelectPanel into dropdown","Currently the image select panel is in a dialog. It also needs to be able to work in a dropdown.",4 DM-5701,"Create toy composite (AST/GWCS) model with supported components","To help us evaluate WCS options, we need to create a relatively complicated composite model in AST and GWCS, using a few models currently available within the existing packages. A minimal composite model to test these things would include: * FITS linear transform * ccd distortion * optical model * FITS TAN WCS The middle steps do not need to be realistic models, just something that we can use to compare AST's and GWCS's respective interfaces and capabilities for creating the composite model, and test for differences in their results. We can then use this model to evaluate performance when run on different numbers of pixels.",4 DM-5702,"Create a new model in AST/GWCS to represent a complex distortion","Using lessons learned from DM-5701, create a more complex distortion model that cannot be represented from the basic models in GWCS or AST. A good example for this might be a rapidly varying sinusoidal tree-ring-like function that is not well represented by the standard polynomial basis functions. This will test our ability to extend each framework with new models that have not yet been decided on. Once completed, we could plug this back into the composite model in DM-5701.",8 DM-5703,"Evaluate performance of AST/GWCS over a range of numbers of pixels","Once we have a composite distortion model from DM-5701, evaluate the performance of AST and GWCS over a range of numbers of pixels, likely from ~100 through full-CCD (4k^2). As part of this process, we will try to determine whether there is a way to efficiently warp images/postage stamps using python-only models in GWCS and whether bottlenecks could be worked around via optimizations in cython.",8 DM-5704,"add cloudbees-folder support to puppet-jenkins ",NULL,6 DM-5715,"Produce document describing DRP parallelization use cases","At various times in the past few months I've promised [~gpdf], [~petravick], [~kooper], and probably a few others a document describing the parallelization needs for DRP in greater detail. My understanding of the plans for the eventual DRP probably good enough to do this well now, and is unlikely to improve further in the near future (as that will require algorithmic research). This needs to be prioritized with my other responsibility for documents that describe the DRP system in other ways, most of which are oriented towards scientists and science pipelines developers. The document on this ticket is essentially the description that would matter the most for the process control middleware team.",6 DM-5716,"UI Consistency","There is a need to go though the entire ui and document inconsistencies with the old UI.",4 DM-5717,"Firefly Result view architecture/component","The result view architecture needs to be written. * A meeting needs to happen with David, Gregory, Xiuqin, Trey, and Tatiana to discuss this. * Trey, Loi, and Tatiana should have a design meeting. It should support some or all of the following ideas: * A search defining a new results view type * A search adding to an existing result view * A search replacing the results - any cleanup needs to happen. * Some sort of controller that know which view should show and which view can be shown * New search panels easily added. Maybe the html file defines which are visible",10 DM-5719,"Verification Cluster, Object Store Procurement","Strategy design with pipeline and deployment teams. Discussions of service description and levels of support. Sufficient design to lead to procurement. Discussions with vendors. Quote selection. Budget tracking. Quote submission to finance. GCO follow up questions. OBFS follow up questions. Finance follow up questions. Overall tracking of purchase progression.",12 DM-5720,"JIRA fixes","This tracks SPs spent on JIRA requests. ",2 DM-5722,"Add table client-side sorting","Convert gwt's client-side sorting to javascript.",4 DM-5723,"make sure table can be resized properly","Test table to make sure it can be resized under a variety of layout.",2 DM-5725,"attend the weekly meeting with UIUC camera team","While Tatiana is the assignee of this ticket, Xiuqin and Gregory participate this weekly telecon semi-regularly to lend support. ",2 DM-5726,"attend the weekly meeting with UIUC camera team (May 2016)","Tatiana will attend the weekly meeting. Xiuqin and Gregory also attends when needed. ",2 DM-5728,"Create django project and initial dashboard app","This ticket captures the steps to create the django project for SQUASH, its configuration and the dashboard app http://sqr-009.lsst.io/en/latest/ The planned tasks are: - Implement the ``Dataset``, ``Visit`` and ``Ccd`` tables in the django ORM layer, as a minimum set of tables for the dashboard app - Prototype home page and dashboard pages ",5 DM-5729,"Config.loadFromStream suppresses NameError","Within a config override file being executed via {{Config.load}} or {{Config.loadFromStream}}, using a variable that hasn't been defined results in a {{NameError}} exception, but this is silently suppressed and the user has no idea the following overrides have not been executed.",1 DM-5734,"Fix the issues in the server side and the client side introduced by FitsHeaderViewer 's work","* The testing data ""table_data.tbl"" in the testing tree was accidentally moved. It should be added back so that IpactTableTest.java can run. * The request in JsontableUtil was mistakenly moved out from the tableModel by the the line * {code} * if (request != null && request.getMeta().keySet().size()>1) { tableModel.put(""request"", toJsonTableRequest(request)); } {code}. The meta can be null but the request is not null, the request should be put into the TableModel. ",1 DM-5738,"Move Camera creation out of CameraMapper base class","With the new cameraGeom, it's considered desirable that each camera be able to define the serialization format for its static camera data. Despite this, it's still the base CameraMapper that does loads it (at least for most cameras), going through a circuitous chain of policy files, obs_* package paths, and Python code. It'd be vastly simpler for each mapper to simply build the Camera object and assign it to {{self.camera}} in its own {{\_\_init\_\_}} method (most would simply delegate all the work to {{afw.cameraGeom.makeCameraFromPath}}). We could then remove all the camera entries from the PAF files and make it much easier to follow the logic. Eventually, I think we need to be storing at least some components of the camera definition in the data repository (or something like a calibration repository associated with it), and that would require giving the mapper access to a partially-constructed butler when its time to build the camera. But we can save that for another day. ",2 DM-5739,"--clobber-config modifies input rerun","Using the {{--clobber-config}} option in a child butler repository can cause changes in the parent repository, as we try to rename files to back them up in the parent repository. This is a critical bug because it can cause pipeline outputs to be unexpectedly modified. It should be easy to fix, as it's just a matter of checking whether the files to be renamed backed up are in the output repository. This was originally reported as https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1341 ",1 DM-5740,"Create and deploy common Ansible roles for ELK","Create roles and deploy to Ansible Galaxy. These are common roles for cloud-init (and any other future cloud dependencies), java (openjdk-jdk) and an editors role.",3 DM-5741,"Create and deploy Elasticsearch and Kibana Ansible roles","Create roles and deploy to Ansible Galaxy.",3 DM-5742,"Create and deploy an ELK system","Create a Vagrant configuration and Ansible role to configure and combine Elasticsearch, Logstash and Kibana (ELK).",36 DM-5743,"Create and deploy Logstash, Fluentd and Riemann Ansible roles","Create roles and deploy to Ansible Galaxy. Create a role to combine all the individual projects together.",3 DM-5744,"Create packer automation for ELK","Build packer automation to create machine images to use for the ELK system.",6 DM-5745,"Implement ingestion code for the QA results","The initial database model was implemented in DM-5728 and outputs of the QA analysis code are being produced by the work described in http://dmtn-008.lsst.io/en/latest/ In this ticket we plan to implement and API for listing and creating jobs, metrics and measurements so that a job or the QA analysis code can register this information in the dashboard app.",4 DM-5746,"Build parallel DCR simulator using GalSim","The result of DM-4899 was a simulation tool called StarFast that can quickly make simulated images with realistic Differential Chromatic Refraction. This ticket is to build an equivalent simulator using GalSim to check the accuracy of results and benchmark speed and memory usage. ",6 DM-5747,"SQUASH dashboard prototype design","SQUASH dashboard prototype design is described here http://sqr-009.lsst.io/en/latest/",8 DM-5748,"Upgrade mpi4py to latest upstream","[mpi4py|https://bitbucket.org/mpi4py/] version 2.0 was released in October 2015 with a number of changes. We should upgrade. When upgrading, we should check whether it contains a proper fix for DM-5409 and, if not, file a bug report upstream. This issue should not be addressed until we have proper test coverage on code which uses mpi4py (DM-3845).",1 DM-5750,"Integration of Django and bokeh server",NULL,5 DM-5753,"XCode 7.3 can not link indirect dependencies that use @rpath","With XCode 7.3 on OS X we have difficulties resolving indirect dependencies when those dependencies are referenced using {{@rpath}}. This can be seen with Qserv: {code} Linking shared object build/libqserv_common.dylib ld: file not found: @rpath/libboost_system.dylib for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) {code} where {{libboost_system}} is being loaded via {{libboost_thread}}: {code} $ otool -L $BOOST_DIR/lib/libboost_thread.dylib /Users/timj/work/lsstsw/stack/DarwinX86/boost/1.59.lsst5+fbf04ba888/lib/libboost_thread.dylib: @rpath/libboost_thread.dylib (compatibility version 0.0.0, current version 0.0.0) @rpath/libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1) {code} This problem is also found when doing a {{conda}} build of the stack because in {{conda}} all shared libraries are modified on creation to reference other libraries via the {{@rpath}} mechanism. This bug has been reported to Apple as [rdr://25313838|http://www.openradar.me/25313838] and a [Chromium bug report|https://bugs.chromium.org/p/chromium/issues/detail?id=597459] indicates that the fix is to simply ensure that {{-L}} directives include a trailing slash. ",4 DM-5756,"Update Scons to v2.5.0","Scons 2.5.0 came out over the weekend. There were many fixes to the dependency determination code. The next version of Scons is intended to be 3.0 which will be the first version to support Python 3. Since we fully intend to switch to Python 3.0 in the summer it is prudent for us to ensuer that 2.5.0 works fine before switching to 3.0.0 so that we do not get confused as to why there is breakage in jumping straight to 3.0.0.",2 DM-5757,"FitsHeader's resize and sorting","DM-4494 has merged to the dev. However, there are still two issues remained: * Resize the popup with tabs does not work * Sorting is depending on the BasicTable's sorting",1 DM-5759,"TabPanel needs a way to keep it state between renders","The TabPanel and CollapsiblePanel loses its state when it is re-rendered. It is going to have to have a way keeps it state. Therefore it needs an option to take an ID and keep it state in the store. Use case- tabs of tables then the image plot goes to expanded mode. The table tabs gets reset to the first one. ",4 DM-5760,"XYPlot needs to be expandable","Make XYPlot expandable",2 DM-5761,"XYPlot should support selecting columns from a table","There are should be a way to display all the information about table columns in a table and allow user to choose a column using this table.",10 DM-5762,"XYPlot: Optimize decimated plot aspect ratio","Currently decimation process assumes aspect ration 1. For decimated plots, the displayed area size (or user supplied value) needs to be used as an aspect ratio to approximate square bins. - When aspect ratio changes, decimation process needs to be redone. - To avoid server calls on resize, disallow flexible aspect ratio for decimated data. ",6 DM-5763,"XYPlot: decimation options","User needs to be able to control number of bins and bin size.",3 DM-5764,"XYPlot: separate density plot from scatter plot","At the moment we display data as scatter plot, when the number of points does not exceed decimation limit, and as density plot when the number of points does exceed this limit. Scatter plot and density plot should be separate charts. These are the reasons: - User should be able to create density plot with any number of points - Chart type and display might be different for density plot in the future - Scatter plot look should not change when the number of points exceeds decimation limit - Scatter plot should support errors in the future",5 DM-5765,"Remove unneeded imports in SConstruct","There's an outstanding pull request from an external contributor (Miguel de Val-Borro) [here|https://github.com/lsst/sconsUtils/pull/9] that makes some minor improvements to sconsUtils by cleaning up the imports. Somebody should review and (if appropriate) merge it. (Or, at least, reply to our community!)",1 DM-5766,"Implement spatial exposure selection task","Once DM-3472 lands, it will be possible to write an image selection task that uses the SQLite 3 database produced by {{IndexExposureTask}} (from [daf_ingest|https://github.com/lsst/daf_ingest]) to search for exposures overlapping a region (in particular, the spatial extent of a coadd patch). The {{_rtree_search}} method in {{test_index_exposure.py}} (also from daf_ingest) has an example of how to perform spatial queries quickly. I was originally scheduled to do something in this space, but Paul mentioned that he had plans to refactor the image selection tasks already, and is much more familiar with the pipeline side of things than I am. Therefore, I'm handing off the implementation of the pipeline task mentioned in DM-3472 to him.",6 DM-5767,"Create custom basic coaddition code","Create script to do the following: * Takes a list of DECam exposure numbers * for each CCD, loads the corresponding calexps * creates a naive pixel-by-pixel coadd of the underlying images * Possibly either ANDs or ORs the masks (though perhaps not necessary) * Either sums the expusure time info from the headers, or averages them, depending on whether the images were normalised to exposure times or not * write the corresponding images out as coadded fits",1 DM-5768,"Coadd CPB exposures","Identify sets of DECam exposures from the CBP run and feed them to the coaddition script created in DM-5767. This will need to be redone each time a reprocessing is done as the script will run on calexps. I will do it once now, and then again after DM-5465 is completed to satisfactory levels.",1 DM-5769,"Write spot visualisation snippets","Write some snippets to aide in the processing and visualisation of the CBP data/analysis. Essentially, write some helper functions that you can throw sections of images at to help look at the shape of the CBP spots, as ds9 isn't great ideal this. Some nice features would be: A function that takes a list of images or arrays, and plots them side-by-side, which provides some intelligent options for the stretches, and optionally stretches each image as is best for it, or ties them all to be the same. This would be as 2D colour plots. A function that takes part of an image and displays it as a colour-graded surface. A function that takes part of an image and displays it as a 3D bar-chart (as in ROOT, but without using ROOT because there is already enough evil in the world)",2 DM-5770,"Investigate image processing for feature enhancement","Whilst looking at an individual spot from the CBP on DECam I noticed a weird feature, and upon further investigation, several more, though these were very hard to see. This ticket is to investigate what image processing techniques will make these hard-to-see features pop out so that they can be examined more closely.",2 DM-5771,"Update config files","DM-46921 and DM-5348 changed ProcessCcd to the point where past config files are no longer valid as stuff has moved a lot (see https://community.lsst.org/t/backward-incompatible-changes-to-processccdtask-and-subtasks/581) This ticket is to go through past configs and create a new config file to reproduce the reductions done, or at least make something sensible come out the end of processCcd",2 DM-5773,"Firefly API plan and decision","We need a plan for all the Firefly APIs development in the new React/Redux based JS framework, including JS API and Python API. - Backward compatibility - Syntax format for JS API - Syntax format for Python API - Schedule - convert the existing API first - list of new ones to be added, when ",2 DM-5775,"Change the TabPanel.jsx and TabPanel.css's properties to allow its children can be resizable","When an outside container is resizable (using css properties: resize: 'both', overflow: 'auto'...), in order for the child inside the container to be resizable, the child has to specify its height and width properties using percentage format (height: 90%, width:100%). When the TabPanel is used, the table is put on TabPanel. The table needs to access the size information of the outside container, ie,, the grandparent's width and height. The TabPanel has to pass the height and width to its child component. Without specifying the height and width in the TabPanel, by default, the auto is used. When the width (height) is auto, it allows to use the child's width (height). However, the child replies on the parent to provide such information. When this circular relations occur, the default size of the child is used. That is why the table component forever has 75px when it was put in the TabPanel. To be able to resize with the outside (root) contains all the ancestors of the component have to specify the width and height explicitly. ",1 DM-5778,"Document Configurable concept","The Configurable concept (a callable that takes a config as an argument) is a fairly important one in pex_config as the guts behind RegistryField and ConfigurableField, and it's mentioned several times in pex_config's documentation, but it doesn't seem to be directly documented itself.",1 DM-5780,"Assist schandra with ts_wep Luigi implementation","Assist [~schandra] with an initial implementation of his workflow using Luigi.",1 DM-5781,"Port Data set info converter, part2","Part 2 includes: * 3 color * clean up on plot fail * clean up image in general * better row highlighting * other types of data layout (a FC type view) * artifacts (maybe part 3) * When image is small, like zoom it down 1/16x, behavior of selecting an image is not consistent. Clicking on an edge will select one, but will not work on another.",10 DM-5782,"Include obs_cfht, obs_decam in lsst-dev shared stack","The shared stack on {{lsst-dev}} provided in DM-5435 does not contain the {{obs_cfht}} or {{obs_decam}} camera packages. Please add them.",1 DM-5784,"Port region serializer and data structures from GWT","The region serializer in: firefly/src/firefly/java/edu/caltech/ipac/util * RegionFactory.java Region container data structures files in : firefly/src/firefly/java/edu/caltech/ipac/util/dd * ContainsOptions.java * Global.java * RegionFileElement.java * RegParseException.java * Region.java * RegionAnnulus.java * RegionBox.java * RegionBoxAnnulus.java * RegionCsys.java * RegionDimension.java * RegionEllipse.java * RegionEllipseAnnulus.java * RegionFont.java * RegionLines.java * RegionOptions.java * RegionPoint.java * RegionText.java * RegionValue.java Note - do not port CoordException, there are other ways to do this.",8 DM-5787,"Add support for SGE","Jean Coupon has requested support for SGE in ctrl_pool.",1 DM-5791,"Why is doSelectUnresolved an argument?","The {{run}} method in the {{PhotoCalTask}} has an argument that selects whether to use the extendedness parameter to select objects for photometric calibration. This is a good idea, but it should be configurable, I think. ",1 DM-5792,"Support artifacts ","For now this is the artifacts for WISE images. We should look at the possibilities to generalize this. ",8 DM-5793,"Convert Mask support",NULL,6 DM-5794,"Support image and drawing layer subgrouping",NULL,8 DM-5795,"Add Python properties for getters and setters in afw::geom and shapelet","I'm adding properties via Swig %extend in much of afw::geom right now, because: - I think we've all agreed this is something we want, even if we haven't agreed how much effort we want to put into it. - I'm getting annoyed writing lots of parentheses for these getters and setters on DM-5197. - I can get this done in a couple of hours on a weekend, so I don't need a T/CAM to give me permission to spend my own time on it :) ",1 DM-5797,"Using 'CONSTANT' for background subtraction fails","Running processCcd (on a DECam file) with the following in the config file: {code} config.charImage.repair.cosmicray.background.algorithm='AKIMA_SPLINE' config.charImage.background.algorithm='CONSTANT' config.charImage.detectAndMeasure.detection.tempLocalBackground.algorithm='CONSTANT' config.charImage.detectAndMeasure.detection.background.algorithm='CONSTANT' config.calibrate.detectAndMeasure.detection.tempLocalBackground.algorithm='CONSTANT' config.calibrate.detectAndMeasure.detection.background.algorithm='CONSTANT' {code} fails, and throws the following: {code} Traceback (most recent call last): File ""/home/mfisherlevine/lsst/pipe_tasks/bin/processCcd.py"", line 25, in <module> ProcessCcdTask.parseAndRun() File ""/ssd/lsstsw/stack/Linux64/pipe_base/2016_01.0-6-g7751869+8/python/lsst/pipe/base/cmdLineTask.py"", line 450, in parseAndRun resultList = taskRunner.run(parsedCmd) File ""/ssd/lsstsw/stack/Linux64/pipe_base/2016_01.0-6-g7751869+8/python/lsst/pipe/base/cmdLineTask.py"", line 199, in run resultList = mapFunc(self, targetList) File ""/ssd/lsstsw/stack/Linux64/pipe_base/2016_01.0-6-g7751869+8/python/lsst/pipe/base/cmdLineTask.py"", line 324, in __call__ result = task.run(dataRef, **kwargs) File ""/ssd/lsstsw/stack/Linux64/pipe_base/2016_01.0-6-g7751869+8/python/lsst/pipe/base/timer.py"", line 118, in wrapper res = func(self, *args, **keyArgs) File ""/home/mfisherlevine/lsst/pipe_tasks/python/lsst/pipe/tasks/processCcd.py"", line 170, in run doUnpersist = False, File ""/ssd/lsstsw/stack/Linux64/pipe_base/2016_01.0-6-g7751869+8/python/lsst/pipe/base/timer.py"", line 118, in wrapper res = func(self, *args, **keyArgs) File ""/home/mfisherlevine/lsst/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py"", line 298, in run background = background, File ""/ssd/lsstsw/stack/Linux64/pipe_base/2016_01.0-6-g7751869+8/python/lsst/pipe/base/timer.py"", line 118, in wrapper res = func(self, *args, **keyArgs) File ""/home/mfisherlevine/lsst/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py"", line 356, in characterize image -= estBg.getImageF() File ""/home/mfisherlevine/lsst/afw/python/lsst/afw/math/mathLib.py"", line 5788, in getImageF return _mathLib.Background_getImageF(self, *args) lsst.pex.exceptions.wrappers.InvalidParameterError: File ""src/math/Interpolate.cc"", line 61, in std::pair<std::vector<double>, std::vector<double> > lsst::afw::math::{anonymous}::recenter(const std::vector<double>&, const std::vector<double>&) You must provide at least 1 point {0} File ""src/math/BackgroundMI.cc"", line 196, in void lsst::afw::math::BackgroundMI::_setGridColumns(lsst::afw::math::Interpolate::Style, lsst::afw::math::UndersampleStyle, int, const std::vector<int>&) const setting _gridcolumns {1} lsst::pex::exceptions::InvalidParameterError: 'You must provide at least 1 point {0}; setting _gridcolumns {1} {code}",2 DM-5798,"Pass butler to ref loader","The design of the indexed reference catalogs requires a butler to be sent to the loader. This requires passing the butler down through the chain of subtasks from the parent command line task. In this case, I believe only calibrateTask constructs sub-tasks that requires a reference catalog. This will also require moving the loader and indexer to meas_astrom, otherwise it will introduce a circular dependency.",4 DM-5799,"Asinh stretch algorithm corerction","in DM-2634, the Asinh stretch algorithm was implemented, but the behavior was not quite right. We need to figure out the issue and make it right. One possibility is that the understanding the relationship of zero point and black point, maximum point and white point. ",8 DM-5800,"TabPanel: Tab titles need to shrink to accommodate a large number of tabs.","Should convert GWT's logic over to TabPanel. - shrink title as needed. - show full title on mouse over",4 DM-5802,"Cmake in mariadbclient finds wrong libz","When building mariadbclient, cmake identifies libz from a separate python installation than the one setup to run the stack. I have an anaconda installation on the disk, and a miniconda installation set up specifically for the lsst stack. During the building process CMake for some reason finds the alternate libz associated with that python installation.",1 DM-5803,"fetchUrl is not handling post requests correctly.","Parameters are not sent to the server when requests are posted via fetchUrl.",2 DM-5804,"Use aperture-corrected aperture flux in validate_drp","Shift from PsfFlux flux/magnitude to aperture-corrected aperture-based mag/flux measurements for calculating photometric repeatibility.",1 DM-5805,"Improve star/galaxy separation for validate_drp","Improve the star/galaxy separation for validate_drp. Many of the LSST SRD KPMs are defined for bright, isolated stars. There is clear evidence that galaxies are being included in current runs (they have significantly higher photometric scatter at the same mag|SNR). Improved star/galaxy separation will help generate better numbers Stretch goal: Include additional informative plots about how well the `extendedness` value in the catalogs is successfully separating stars and galaxies.",1 DM-5806,"Add a paging bar to ImageMetaDataToolbarView","Add a paging bar similar to the one for table to the ImageMetaDataToolbarView. This pages images instead of rows.",4 DM-5808,"Ensure that variance plane in calexps is unchanged HSC⟷LSST","Per discussion in HSC telecon 2016-04-19.",1 DM-5810,"Update imageDifferenceTask to cast template ids and use ObjectSizeStarSelector","A couple recent changes to the stack break imageDifferenceTask. Requires updates to only a few lines. While I'm updating it to reflect the star selector API, I'm also changing the default star selector from SecondMoment to ObjectSizeStarSelector (which I learned today is what the stack has been using by default for a while). ",1 DM-5819,"Incorporate Price suggestions to make `validate_drp` faster","Increase the loading and processing speed of {{validate_drp}} following suggestions by [~price] 1. Don't read in footprints Pass {{flags=lsst.afw.table.SOURCE_IO_NO_FOOTPRINTS}} to {{butler.get}} 2. Work on speed of calculation of RMS and other expensive quantities. Current suggestions: a. {{calcRmsDistances}} b. {{multiMatch}} c. {{matchVisitComputeDistance}} d. Consider boolean indexing {code} objById = {record.get(self.objectKey): record for record in self.reference} to: objById = dict(zip(self.reference[self.objectKey], self.reference)) {code} Note that while this ticket will involve work to reduce the memory footprint of the processing, it will not cover work to re-architect things to enable efficient processing beyond the memory on one node.",2 DM-5820,"3 color and FITS header clean up","There are some issues with three color when not using all three bands (i.e. using on green and blue): * Mouse readout is not labeled correctly * FITS head popup does not come up Other FITS header popup issues: * If file size is too big then the text is wrapping * On safari, the resizable indicator in on every cell ",6 DM-5821,"Intermittent fault building ci_hsc through Jenkins","Occasionally (see e.g. [here|https://ci.lsst.codes/job/stack-os-matrix/label=centos-7/10437//console] and [here|https://ci.lsst.codes/job/stack-os-matrix/label=centos-6/9594//console]) the {{ci_hsc}} job in Jenkins fails, reporting: {code} RuntimeError: dictionary changed size during iteration {code} The fault seems to be intermittent. Please fix it.",3 DM-5822,"Afw fails unit test for convolve depending on compiler optimisation level","On OSX 10.11.4 with Apple LLVM version 7.3.0 (clang-703.0.29) afw fails {{test/convolve.py}} with the following error when either {{-O0}} or {{-O1}} is enabled but works fine for {{-O2}} and {{-O3}}. {code:bash} tests/convolve.py .....FF/Users/pschella/Development/lsst/code/afw/python/lsst/afw/image/testUtils.py:283: RuntimeWarning: invalid value encountered in isnan nan0 = np.isnan(filledArr0) /Users/pschella/Development/lsst/lsstsw/miniconda/lib/python2.7/site-packages/numpy/lib/ufunclike.py:113: RuntimeWarning: invalid value encountered in isinf nx.logical_and(nx.isinf(x), ~nx.signbit(x), y) /Users/pschella/Development/lsst/lsstsw/miniconda/lib/python2.7/site-packages/numpy/lib/ufunclike.py:176: RuntimeWarning: invalid value encountered in isinf nx.logical_and(nx.isinf(x), nx.signbit(x), y) F.F... ====================================================================== FAIL: testSpatiallyVaryingAnalyticConvolve (__main__.ConvolveTestCase) Test in-place convolution with a spatially varying AnalyticKernel ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/convolve.py"", line 437, in testSpatiallyVaryingAnalyticConvolve rtol = rtol) File ""tests/convolve.py"", line 290, in runStdTest self.runBasicConvolveEdgeTest(kernel, kernelDescr) File ""tests/convolve.py"", line 317, in runBasicConvolveEdgeTest doVariance = True, rtol=0, atol=0, msg=msg) File ""/Users/pschella/Development/lsst/code/afw/python/lsst/afw/image/testUtils.py"", line 201, in assertMaskedImagesNearlyEqual testCase.fail(""%s: %s"" % (msg, ""; "".join(errStrList))) AssertionError: basicConvolve(MaskedImage, kernel=Spatially Varying Gaussian Analytic Kernel using brute force) wrote to edge pixels: image planes differ: maxDiff=1.09176e+38 at position (73, 18); value=-1.09176e+38 vs. 2825.0; NaNs differ ====================================================================== FAIL: testSpatiallyVaryingDeltaFunctionLinearCombination (__main__.ConvolveTestCase) Test convolution with a spatially varying LinearCombinationKernel of delta function basis kernels. ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/convolve.py"", line 556, in testSpatiallyVaryingDeltaFunctionLinearCombination rtol = rtol) File ""tests/convolve.py"", line 290, in runStdTest self.runBasicConvolveEdgeTest(kernel, kernelDescr) File ""tests/convolve.py"", line 317, in runBasicConvolveEdgeTest doVariance = True, rtol=0, atol=0, msg=msg) File ""/Users/pschella/Development/lsst/code/afw/python/lsst/afw/image/testUtils.py"", line 201, in assertMaskedImagesNearlyEqual testCase.fail(""%s: %s"" % (msg, ""; "".join(errStrList))) AssertionError: basicConvolve(MaskedImage, kernel=Spatially varying LinearCombinationKernel of delta function kernels using brute force) wrote to edge pixels: image planes differ: maxDiff=9.06659e+36 at position (75, 29); value=9.06659e+36 vs. 2865.0 ====================================================================== FAIL: testSpatiallyVaryingGaussianLinerCombination (__main__.ConvolveTestCase) Test convolution with a spatially varying LinearCombinationKernel of two Gaussian basis kernels. ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/convolve.py"", line 523, in testSpatiallyVaryingGaussianLinerCombination rtol = rtol) File ""tests/convolve.py"", line 290, in runStdTest self.runBasicConvolveEdgeTest(kernel, kernelDescr) File ""tests/convolve.py"", line 317, in runBasicConvolveEdgeTest doVariance = True, rtol=0, atol=0, msg=msg) File ""/Users/pschella/Development/lsst/code/afw/python/lsst/afw/image/testUtils.py"", line 201, in assertMaskedImagesNearlyEqual testCase.fail(""%s: %s"" % (msg, ""; "".join(errStrList))) AssertionError: basicConvolve(MaskedImage, kernel=Spatially Varying Gaussian Analytic Kernel with 3 basis kernels convolved using brute force) wrote to edge pixels: image planes differ: maxDiff=1.22472e+38 at position (74, 3); value=-1.22472e+38 vs. 2878.0; NaNs differ ====================================================================== FAIL: testTicket873 (__main__.ConvolveTestCase) Demonstrate ticket 873: convolution of a MaskedImage with a spatially varying ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/convolve.py"", line 623, in testTicket873 rtol = rtol) File ""tests/convolve.py"", line 290, in runStdTest self.runBasicConvolveEdgeTest(kernel, kernelDescr) File ""tests/convolve.py"", line 317, in runBasicConvolveEdgeTest doVariance = True, rtol=0, atol=0, msg=msg) File ""/Users/pschella/Development/lsst/code/afw/python/lsst/afw/image/testUtils.py"", line 201, in assertMaskedImagesNearlyEqual testCase.fail(""%s: %s"" % (msg, ""; "".join(errStrList))) AssertionError: basicConvolve(MaskedImage, kernel=Spatially varying LinearCombinationKernel of basis kernels with low covariance, using brute force) wrote to edge pixels: image planes differ: maxDiff=3.19374e+38 at position (1, 46); value=3.19374e+38 vs. 2774.0 ---------------------------------------------------------------------- Ran 13 tests in 43.252s FAILED (failures=4) The following tests failed: /Users/pschella/Development/lsst/code/afw/tests/.tests/convolve.py.failed 1 tests failed scons: *** [checkTestStatus] Error 1 scons: building terminated because of errors. {code}",2 DM-5823,"ECL_B1950 coordinate was not defined correctly","The CoordSys.js defined ECL_B1950 incorrectly. When I was testing WebGrid, the grid lines for Ecliptic B1950 were not right. Looked further, it was caused by wrong equinox value in its definition.",1 DM-5825,"Add CI tests for obs_lsstSim","I propose: 1. Create {{testdata_lsstSim}}. This will be based on 12 images from the current Twinkles Run 1 (or pre-Run 1): 2 epochs each of 6 filters. (/) 2. Add an optional test method to {{obs_lsstSim}} that runs if {{testdata_lsstSim}} has been declared. This is the way these tests are set up for the other {{obs_*}} packages. 3. Add {{testdata_lsstSim}} dependency to {{lsst_ci}} (which already depends on {{obs_lsstSim}}). This will then be run every time a full standard default Jenkins build is processed.",2 DM-5827,"Compare LSST and HSC pipelines through through multi-band coadd processing","Continue the work described in DM-5301 through the standard ""multi-band"" coadd processing workflow. Performing an end-to-end comparison of the stacks will not be possible until {{meas_mosaic}} is fully operational on LSST (DM-2674). However, until that point is reached, comparisons are still possible by either: * Shepherding data through {{meas_mosaic}} and coaddition on HSC, then performing further processing and measurement using the LSST stack; * Omitting {{meas_mosaic}} from the workflow altogether and performing end-to-end comparisons of the stacks without mosaicking. Obviously, neither of these will ultimately be adequate, but they should enable early identification of any major issues.",15 DM-5829,"Create outline of Level 3 ConOps","Create an outline of the sections of the Level 3 ConOps document",2 DM-5830,"Level 3 requirements flowdown","Document the flowdown of Level 3-related requirements from SRD, LSR, OSS, and DMSR.",3 DM-5831,"SUIT requirement flowdown","go through the original requirement of SUIT, put them in the categories: done, Tier1, Tier2 ",6 DM-5832,"LSE-140 post-CCB implementation","Following CCB approval of LSE-140, perform minor document work required for full implementation (application of standard cover page, change log, etc.).",2 DM-5833,"SUIT design diagramming","Prepare initial set of SysML diagrams of the SUIT's relationship to other system components.",4 DM-5834,"Prepare requirements and design for Fall 2016 SUIT deployments","Prepare functional and quantitative requirements and the SUIT-centric elements of design for the planned Fall 2016 SUIT deployments (SDSS Stripe 82 and WISE).",10 DM-5835,"Prepare a draft of the SUIT deployment timeline","Prepare a draft schedule, with some detail for 2016-2017, for deployments of the SUIT into (test) production, including the datasets that will be served.",2 DM-5836,"access to NCSA Nebular to setup servers for SUIT deployment","Get three hosts in NCSA nebular system to deploy the current Firefly application. The goal is workout the possible issues and identify the software needed to be installed for the hosts. Clarify which team is responsible to install what third-party software packages.",4 DM-5837,"Document pipe_drivers","Please provide a minimal level of documentation for {{pipe_drivers}}, to include: * A {{doc}} directory with the usual content so that docstrings get generated by Doxygen; * A package overview; * All docstrings should be appropriate for parsing by Doxygen (ie, should start with {{""""""!}} where necessary).",2 DM-5839,"horizon console interface broken","It appears that at some point in the last few months the horizon console interface has stopped working. I am still able to access the console log output via the API/CLI.",1 DM-5840,"instance limit low vs available cores","The LSST project is currently at 81/100 instances but there are over 200 cores unused. Is it possible to increase the instance limit or are we being encouraged to use large instance flavors?",1 DM-5841,"unable to list nebula lsst project users","Currently, [with some difficulty] it is possible to discover the {{user_id}} that created an instance (might be possible for other resources as well) but it is not possible to map this back to a username / person. This can make it difficult to 'self police' instances. The administrative API endpoints are not publicly accessible and I doubt any end user has the appropriate permission. ",1 DM-5844,"automate deployment of qa dashboard server and database instance","Add a qa server + rds instance to the terraform configuration for the jenkins-demo sandbox for development purposes. It may make sense to split this off to be an independent sandbox but that is very easy to do, if needed.",20 DM-5845,"ci_hsc fails with ""too many open files""","For example, with thanks to [~wmwood-vasey]: {code} ci_hsc: master-g78db638f21 .....................................................................................ERROR (207 sec). *** error building product ci_hsc. *** exit code = 2 *** log is in /Users/wmwv/lsstsw/build/ci_hsc/_build.log *** last few lines: ::::: [2016-04-25T19:25:59.824660Z] jobs.run(postfunc = jobs_postfunc) ::::: [2016-04-25T19:25:59.824699Z] File ""/Users/wmwv/lsstsw/stack/DarwinX86/scons/2.3.5/lib/scons/SCons/Job.py"", line 113: ::::: [2016-04-25T19:25:59.824709Z] postfunc() ::::: [2016-04-25T19:25:59.824752Z] File ""/Users/wmwv/lsstsw/stack/DarwinX86/scons/2.3.5/lib/scons/SCons/Script/Main.py"", line 1294: ::::: [2016-04-25T19:25:59.824767Z] SCons.SConsign.write() ::::: [2016-04-25T19:25:59.824808Z] File ""/Users/wmwv/lsstsw/stack/DarwinX86/scons/2.3.5/lib/scons/SCons/SConsign.py"", line 109: ::::: [2016-04-25T19:25:59.824816Z] None ::::: [2016-04-25T19:25:59.824869Z] File ""/Users/wmwv/lsstsw/stack/DarwinX86/scons/2.3.5/lib/scons/SCons/dblite.py"", line 116: ::::: [2016-04-25T19:25:59.824878Z] None ::::: [2016-04-25T19:25:59.935601Z] Exception IOError: (24, 'Too many open files', '.sconsign.tmp') in <bound method dblite.__del__ of <SCons.dblite.dblite object at 0x10dfe9c50>> ignored {code} Possibly only happens on OSX?",2 DM-5847,"libxml build issue with mpich on OS X","On OS X with Xcode installed {{mpich}} fails to build because it can not locate the libxml include files: {code} CC topology-xml-libxml.lo topology-xml-libxml.c:17:10: fatal error: 'libxml/parser.h' file not found #include <libxml/parser.h> ^ 1 error generated. {code} with {{pkg-config}} 0.29.1 installed. The problem is that {{configure}} determines that {{libxml-2.0}} is available and is installed into {{/usr}} with a CFLAGS of {{-I/usr/include/libxml2}}. {{configure}} does not itself test whether those parameters are reasonable. With Xcode there are no files installed into {{/usr/include}} and {{clang}} knows to look in specific SDK locations. When {{mpich}} builds it assumes that {{libxml2}} can be found but fails to find it. Strangely, {{pkg-config}} v0.28 does not seem to be able to find {{libxml-2.0}} so there is no issue. One solution is to install the Command Line Tools but it might be more portable to attempt to disable {{libxml2}}. ",2 DM-5848,"Investigate Jupyter internals, interactive widgets","In preparation for linking Jupyter notebooks with Firefly and other SUIT components, read Jupyter documentation. Learn how to build a sample widget or interactive dashboard in the Jupyter framework",2 DM-5849,"Investigate Ginga and Glueviz visualization tools","Ginga and Glue (glueviz) are community visualization tools in Python. Become familiar with the capabilities of both, thinking from the point of view of using Firefly for the display but using Python for many other things.",2 DM-5854,"Java array index out of bound error in VisSeverCommand.java","The class FileFluxCmdJson in VisServerCommand.java is calling {code} String[] res = VisServerOps.getFileFlux(fahAry, pt); {code} However, when the mouse is outside the image, the VisServerOps.getFileFlux(fahAry, pt) returns: {code} new String[]{PlotState.NO_CONTEXT} {code} It is fine for a single band. However, for 2 or 3 bands, the for loop below caused the index out of bound error because res is an array of length=1 and the expected res is an array of length=no of bands. {code} JSONObject obj= new JSONObject(); obj.put(""JSON"", true); obj.put(""success"", true); int cnt=0; JSONObject data= new JSONObject(); for(Band b : state.getBands()) { data.put(b.toString(), res[cnt++]); } data.put(""success"", true); {code} Thus, res\[cnt++\] caused array index out of bound error. To fix this issue, the for loop is changed as below: {code} int cnt=0; JSONObject data= new JSONObject(); Band[] bands = state.getBands(); for (int i=0; i<res.length; i++){ data.put(bands[i].toString(), res[i]); } data.put(""success"", true); JSONArray wrapperAry= new JSONArray(); obj.put(""data"", data); wrapperAry.add(obj); {code} When the mouse is outside the image, the res returns a new String\[\]\{PlotState.NO_CONTEXT\}, it is added to the JSONObject only once. ",1 DM-5857,"Make DipoleFitPlugin mask-safe","The DipoleFitPlugin does not correctly handle bad pixels and other masks/flags. Make it so it does so, and make tests to ensure it does so.",6 DM-5858,"LSST the Docs Production Fall 2016","DM-5404 introduced _LSST the Docs_ as a production platform for continuous documentation delivery. This Epic covers additional improvements to the platform, such as - Implementation of a backup system for LSST the Docs’ DB - Edition and build dashboards at the /v/ and /builds/ directories that help users find the appropriate version of the documentation site. These could be rendered with react from the API. This also serves as a ramping up exercise on UI elements that will be used on the SQuaSH dashboard and DocHub, so learning time has been rolled into the estimate",42 DM-5859,"Table: Add keyboard navigation","- Added arrow up/down to move between rows. - Added page up/down to move between pages. - Fixed table loading mask not showing - Fixed PagingBar rendering more than it should - Fixed annoying StandardView missing unique key warning",2 DM-5860,"Create obs_monocam","Make a package to hold the description of moncam.",10 DM-5863,"Minor tweaks to Cython and pybind11 tech notes","I'll be making some superficial changes to the text of DMTN-13 and DMTN-14 for grammar, while updating links to the python-cpp-challenge repo (which has just moved from my private GitHub to lsst-dm).",1 DM-5864,"Improve Qserv CI using multinode tests","Here's some tracks: 1. Run multinode integration tests during qserv_distrib CI build. In order to do that we could create a qserv_testmultinodes repository containing a build script which would launch multinode tests (for example see travis.yml) I can do this on my side but i'll require a recent version of Docker on the build machine. 2. I'll need your help to do next step: Each time command below succeed: {code:bash} rebuild qserv_distrib {code} publish this build to eups web repository and docker hub by running: {code:bash} # bXXX is the build id and is available at the bottom of rebuild command standard output publish -b bXXX -t qserv-dev qserv_distrib # then create and publish to docker hub image ""qserv/qserv:dev"" # which embed current build products and is used for all Qserv deployment # (bare-metal, openstack, travis, developper machines) $QSERV_DIR/admin/tools/docker/2_build-dev-image.sh {code} As a TODO list, here's what could be done in an additional ticket, in the long term: - run multinode integration test inside Jenkins instead of Travis? * Travis/Gitub integration is very good, so I'm not sure this feature is still an active concern? * on the other hand Travis has to download qserv/qserv:dev at each build, and if there's a timeout here, the build sometime fails. I don't know if this image can be cached in Travis free version? * current procedure doesn't support yet tickets branch accross multiple repositories, (like qserv+xrootd+qserv_testdata for example). Do you think this feature would be easier to implement in Jenkins?",10 DM-5866,"Test lsst.log with pipeline tasks","Try to use {{lsst.log}} instead of {{lsst.pex.logging}} for a few science pipeline tasks, based on log {{u/ktlim/getLogger}} branch and DM-3532. Look into RFC-29.",10 DM-5868,"Literature research on image subtraction algorithms","We need to get a good understanding of where the image subtraction implementation in the stack currently stands. This first requires an up-to-date assessment of the literature, including Becker et al. (2012), and ZOGY (2016). Also, the ""preconvolution"" step.",8 DM-5869,"Assessment of current state-of-the-stack diffim implementation","The existing diffim implementation in the stack defaults to the (2000) version of the Alard/Lupton algorithm. Other recent improvements such as ""pre-convolution"", delta-function basis, model selection via BIC, others, seem to be implemented but are not turned on. We need a good understanding of the existing implementation so we can assess how straightforward it is to implement the ZOGY algorithm in real-space in the stack.",6 DM-5870,"Update testdata_subaru to support calib changes","Merging DM-5124 broke obs_subaru because the test data in testdata_subaru wasn't updated. Fix it.",1 DM-5872,"Incorporate ""Bickerton algorithm"" for detecting & masking satellite trails","In [HSC-1272|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1272], [~bick] proposed an algorithm for detecting and masking satellite trails. This has undergone some review on HSC, but has never been incorporated into an HSC software or data release (and hence is not part of the ""HSC port""). However: the algorithm is certainly relevant to LSST. Please convert it to work with the LSST stack.",5 DM-5874,"Produce document describing flavors of coadds","[~zivezic] has requested a description of the different flavors of coadds, and the tradeoffs between the more experimental optimal coadds and the non-optimal standard ones. I'll try to do this as both a presentation for the DMLT and a DMTN (with essentially the same content). This will include a bit of toy-model simulation work to try to predict some of the tradeoffs; this could make the discussion quite a bit more quantitative and I have an idea for how to do it that is pretty easy.",10 DM-5875,"Propose text for alternate galaxy models in DPDD","Write a paragraph or two describing alternatives to the constrained bulge+disk model currently in the DPDD.",1 DM-5877,"Use Afterburners to clean up aperture correction logic","This issue has several components; I'm combining them into a single issue because they need to be done atomically: - Rewrite the base_ClassificationExtendedness SingleFramePlugin/ForcedPlugin as an AfterburnerPlugin (and remove the old versions). - Move the ""applyApCorr"" subtask out of SingleFrameMeasurementTask and ForcedMeasurementTask, making it instead a subclass of their parent Tasks. - Add afterburner subtask stages to ProcessCcdTask (within DetectAndMeasureTask) and the multiband tasks wherever measurement is currently being run. The afterburner tasks should be run after aperture corrections are measured and/or applied. After these changes, throughout the stack, whenever a MeasurementTask is run, we also run ApplyApCorrTask and AfterburnerTask (in that order), while possibly running MeasureApCorrTask immediately after the MeasurementTask. This may or may not enable significant cleanups in DetectAndMeasureTask (I haven't looked closely). If so, they should be done on this issue. Given all the moving parts, it's important to check that the actual behavior of the pipeline (in the aperture correction and extendedness values) does not change, so it might be useful to start by creating some reference outputs to compare against.",6 DM-5878,"Add chi plots to validate_drp output to compare nominal error","Make histograms of Deltas / nominal error. where ""nominal"" error is that reported by the pipeline. Are they distributed with a sigma=1?",2 DM-5879,"Remove use of Boost smart pointers throughout the Science Pipelines","Replace all use of Boost smart pointers through the stack with their standard library equivalents. This will require an RFC.",10 DM-5880,"Audit use of Boost in the stack and remove it where possible","Consider use of Boost in the stack, and investigate where it can be eliminated by using std library equivalents. Create tickets to remove it, then work on them.",10 DM-5881,"Recover from processCcd refactor","Now that DM-5771 is closed and processCcd runs again, get back to the point where the things are not just not crashing but are actually working correctly (given that this is a non-standard use-case).",10 DM-5884,"Create MySQL account for monitoring","A MySQL account needs to be created in configuration procedure and on existing data on IN2P3 cluster in order to enable ELK access. MySQL password/secret has to be shared accross all containers.",4 DM-5885,"Create a JSON file for monitoring stack","Create a JSON or YAML file with: - Qserv version - libraries/deps version - other idea welcome This will interesting ""GROUP BY"" in monitoring tool (performance for each Qserv or xrootd version for example)",3 DM-5886,"Provide ngmix-based MCMC galaxy fitting","DM-2250 provided for ""simple"" fitting of galaxies in single frame measurement using ngmix. Extend this to fit galaxies using ngmix's MCMC sampling facilities. This may include defining a mechanism to store MCMC samples in source records. This should be available through (an) lsst_apps (or _distrib) based meas_extension package(s).",60 DM-5889,"Suppress gcc warnings about ""unused local typedefs""","We should add {{\-Wno\-unused\-local\-typedefs}} to our gcc options. This cleans up the build significantly, because there's a flood of warnings of this type coming from boost. If we suppress those, it might become possible to notice warnings that we care about.",1 DM-5893,"LSST the Docs Fastly should redirect /en/latest/ to /","Previously we deployed documentation on Read the Docs. By default, Read the Docs would show the master version of documentation on ""/en/latest/"". Many links with that endpoint may already exist. We should configure Fastly to redirect such paths to ""/"".",1 DM-5894,"LSST the Docs Fastly Courtesy Redirects for directory paths","Currently if a user browses {{example.lsst.io/v/main/some-directory}} instead of {{example.lsst.io/v/main/some-directory}} they will receive an error. We should develop a scheme where Fastly can detect that such a path is a directory and redirect to the directory's index.html page.",4 DM-5897,"Robustify coadd","In running the processing from the Twinkles data challenge in DESC we found that it was very easy to use the wrong skymap when making a coadd. Since the coadd code doesn't even make a cursory check that the coordinate system it is using is the same as that of the coaddTempExps, it is very possible to mess this up. Adding a check that the coadd WCS is that of each input tempCoaddExp is would solve this.",2 DM-5898,"Python EUPS package can use $PYTHON","The {{python}} eups package has a script that checks that the python being used is version 2.7. This script can optionally check {{$PYTHON}} rather than the python in the path but I am confused as to what that test is going to do for us. The problem is that {{sconsUtils}} uses {{python}} and most of the shebangs use {{/bin/env python}} (although shebang rewriting on all platforms could help with that). I think the check script should have the {{$PYTHON}} support removed due to excessive confusion. It would also help if the check script worked with python 3 so that the wrong python could be caught.",1 DM-5900,"Create psutil EUPS package","Add the python {{psutil}} package to the stack as {{python_psutil}}.",1 DM-5901,"LTD Keeper: More Robust Edition purges","LTD Keeper needs to purge Fastly when an Edition is rebuilt. Currently the surrogate-key for the build is also used to cover editions. This means that the key needed to purge an edition is the same as that for an build. Hence purging an edition means that the system needs to purge the surrogate key of the previous build. We're seeing situations where the surrogate key that Keeper is purging is not the one that needs to be purged. A more robust configuration would be for each edition to have a stable surrogate-key that can be unambiguously purged. This ticket covers the following work: # Diagnose the issue. # Enable Alembic migrations for Flask (Flask-Migrate) # Add a surrogate-key column to the Edition model # Change the S3 copy rebuild code to change the surrogate-key header # Change the rebuild code to purge based on the edition's surrogate-key.",3 DM-5903,"Finish technical note on galaxy shear experiments","In the review of DM-5447 we decided it made sense for [~jbosch] to take over finishing the technote, in particular providing an introduction and concluion with more context.",4 DM-5904,"Create focus script","In DM-3368, we stripped out the focus calculation since it's not camera-generic, and the scatter/gather isn't necessary for general processing. We need to reinstate the focus calculation in its own scatter/gather script.",2 DM-5905,"Refactor DipoleFitPlugin classification into separate Classification plugin","Currently the new DipoleFitPlugin runs measurement and then classification from a single measurement method. The classification should be moved out to a separate plugin. This will require more information be stored in the measRecord, in order to do the classification separately. Given that complication, evaluate whether this is even worthwhile.",6 DM-5906,"Remove qmeta::QueryId and use global qserv::QueryId","Remove the qmeta::QueryId and use the typedef of QueryId in global/intTypes.h instead. Also try to verify that QueryId is used instead of uint64_t where applicable.",6 DM-5907,"Replace the heap in ScanScheduler with a list.","ScanScheduler is using heaps to order Tasks by chunkId. This makes it difficult to add Tasks to actively running chunks, which causes a significant delay to the start of query execution. Using a list of buckets of Tasks where each bucket is for one chunkId will make it easy to add Tasks to chunks being. Within each bucket it will still be necessary to order Tasks by tables used in the query.",9 DM-5908,"Alter the worker thread pool to allow threads to leave the pool and continue.","There are times when it is desirable for a thread to continue but effectively leave the thread pool and be considered finished by the scheduler. util::ThreadPool needs to modified to do this.",9 DM-5909,"After the first result set is returned, have the thread leave the pool.","When large results are returned from the worker to the czar, the thread should leave the thread pool and the Task should indicate to the scheduler that it is done. ",6 DM-5910,"Add code to the czar to throttle incoming large results.","The czar needs code to limit the number of Tasks sending back large results at any given time.",9 DM-5911,"Fix circular references in Mapper objects","Whilst running tests with pytest and the new file descriptor leak checker it became clear that Mapper objects were not freeing their resources when they were deleted. In particular, the registry objects remained and the associated sqlite database files were opened. This led to pytest running out of file descriptors when large test suites were being executed. The problem turns out to be the dynamically created map functions. These are created as functions (not bound methods) attached to an instance. Since they are not bound methods the instance object (self) has to be passed in to closure. This leads to self containing a reference to a function that contains a reference to self and this prevents the Mapper from ever being garbage collected (leading to all the resources being retained). A short term fix is pass the mappers into the closures using {{weakref}}. Eventually it would be nice to consistently make the {{map_}} items bound methods rather than attaching them as functions but that is beyond the scope of this ticket.",1 DM-5912,"Add ""everything"" scan","Add a low priority ScanScheduler to the worker to handle very slow scans or scans that do not work well on the other schedulers.",3 DM-5913,"Add ability for workers to switch slow queries to the everything scan.","Give the workers the ability to move user queries to the everything scan if they are taking too long to complete a Task or several Tasks.",9 DM-5914,"Document planned implementation of toy model of Lupton(ZOGY)","Develop a better understanding of the planned implementation of ZOGY in real space by implementing the kernel correction in k-space and investigating its characteristics when transformed back into real space. Do this either symbolically (if possible) or numerically in an ipython notebook. First in 1-D, then in 2-D, both assuming a constant kernel. Include documentation in the ipython notebook describing the current understanding of how this will be implemented",6 DM-5915,"Decide how to rework afw:Wcs guts with AST","Following the to-be-written recommendation for DM-4157, we plan to rework the guts of afw:Wcs to use AST. We need to decide how afw:Wcs will use AST, whether as a wrapper or as a complete replacement with AST. The product is a design",8 DM-5916,"Decide how to rework XYTransform/GTransfo guts with AST","We want to better connect our other transforms with the WCS system, which means reworking the guts of XYTransform/GTransfo to work with AST. This could involve making one or both of them a wrapper, complete replacement, or writing a converter that turns our transform object into an AST map or FrameSet. The product is a design",10 DM-5917,"Design an API for the new Wcs and Transform system","We need to design a new API for the WCS/Transform system. This is somewhat independent of the question of how the low-level code is used: we want a clean and simple API that lets the components of the stack create, manipulate, use, and persist the necessary transformations. Related to this question is whether we will still need skyToPixel/pixelToSky or whether the necessary operations with those can be subsumed into some Frame-to-Frame transformation (e.g. pixel-to-pixel or tan-to-tan). The product is an RFC",16 DM-5918,"What transforms do we currently need?","In order to use AST in the stack, we may need to add mappings to it. We also need to be able to describe our transforms at a high level so that we know how to create them. We need a list of the currently necessary transformations (e.g. from afw:wcs, XYTransform, GTransfo and any other relevant stack packages), and some concrete ideas about the kinds of transforms we may need in the future. These should be described in a high-level mathematical manner, independent of our wcs/transform system. This can be informed by DMTN-005 and the requirements section of DMTN-010",4 DM-5919,"Describe our composite mappings and transformation endpoints (Frames)","To use AST in the stack, we need to be clear what our different transformations (AST:Mappings) and endpoints (AST:Frames) are going to be so we can create the chain of transformations (AST:FrameSets) that will be used throughout the stack. This applies to both images and CameraGeom. We may want to produce similar descriptions for other stack objects. This ticket is the high-level Frames equivalent to the mathematical Mapping description in DM-5918. This will help us determine how we can put our current input/output image frames into the new system.",4 DM-5920,"Create DCR metric using new dipole measurement","In order to evaluate DCR correction algorithms we need a metric that defines the severity of DCR in a residual image. This ticket is to run the new dipole measurement task on simulated difference images affected by DCR, and to define a useful metric. The result will be a brief technical note defining the process and the metric, with a few examples.",6 DM-5921,"Clarify how to work with ci_hsc's astrometry_net_data","ci_hsc's {{README.rst}} contains [a note|https://github.com/lsst/ci_hsc/blob/87b6ecb1cc0157cac8dafb356520f49f971bb1ec/README.rst#reference-catalog] on declaring & setting up the included reference catalogue data. I believe this was rendered obsolete by DM-5135, which automatically sets up the reference catalogue when ci_hsc itself is set up. Attempting to follow the documentation therefore produces confusing warning messages, and may break things. Please check if my understanding is correct and, if so, fix the documentation.",1 DM-5922,"Rework camera geometry to use the replacement for XYTransform","As part of overhauling XYTransform we will likely need to replace the way we describe the transformations supported by camera geometry and {{Detector}}. This is likely to include a new way of describing the coordinate frames (e.g. {{PIXEL}}. {{FOCAL_PLANE}} and {{PUPIL}}). If we adopt AST (as seems likely) then these frames will be AST {{Frames}}, the transforms will be AST {{Mappings}} and the collection described by {{Camera}} and {{Detector}} will be one or more AST {{FrameSets}}. An RFC for the redesigned API for camera geometry will be required and this ticket is to implement the resulting design.",8 DM-5923,"Support arbitrary sky rotation angles in StarFast","Currently, if a region of sky is simulated in StarFast the stars must always have the same x,y coordinates (before DCR effects). This ticket is to support arbitrary rotations and offsets of the simulated stars to mimic realistic repeated observations of the same field.",2 DM-5924,"Improve overscan correction","Overscan correction can be improved. Specifically, some systems have sharp discontinuities in the bias section.",6 DM-5925,"Implement fringe correction in ISR","There is an initial implementation of fringe correction in the obs_subaru package. It should be ported and generalized.",6 DM-5926,"networking in strange state for newly created instances","When starting a new instance, occasionally something strange seems to happen with the network setup. The instance will come up but is inaccessible (icmp, ssh). When this happens, the console log shows that a DHCP address was obtained and cloud-init injected ssh-keys, so it isn't a total network setup failure. I have seen this happen a few times in the last couple of weeks but I can't reliably reproduce it. I'm wondering if neutron is logging anything interesting when this happens. This failure mode happened again a few minutes ago with 7adffa82-7221-454c-acfe-5f21cdd34ea8. Which I killed and recreated as instance b6f64981-099b-46e5-a27e-e3694372f447 with the same private IP address. The new instance is accessible as expected.",1 DM-5927,"API errors when trying to start up multiple instances","I am attempting to start up 20 {{m1.medium}} instances without floating IPs to take available of the new instance cap from DM-5840. This consistently fails after starting a few instances with an HTTP 403. {code:java} Error creating OpenStack server: Expected HTTP response code [201 202] when accessing [POST http://nebula.ncsa.illinois.edu:8774/v2/8c1ba1e0b84d486fbe7a665c30030113/servers], but got 403 instead {""forbidden"": {""message"": ""Maximum number of ports exceeded"", ""code"": 403}} {code} Of the instances that do manage to start, most end up in an error state with. {code:java} (openstack) server show 134b69dc-56fc-4249-b92f-e958e561ae3b +--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | error | | OS-SRV-USG:launched_at | None | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | | | config_drive | | | created | 2016-05-02T20:29:54Z | | fault | {'code': 500, 'message': 'No valid host was found. Exceeded max scheduling attempts 3 for instance 134b69dc-56fc-4249-b92f-e958e561ae3b. Last | | | exception: [u\'Traceback (most recent call last):\\n\', u\' File ""/usr/lib/python2.7/site-packages/nova/compute/manager.py"", line 2235, in _do', | | | 'created': '2016-05-02T20:29:57Z'} | | flavor | m1.medium (3) | | hostId | b383eddb06f7a1cc5929e5fa8b6982cc523f5ac1cbe3c9c40120a700 | | id | 134b69dc-56fc-4249-b92f-e958e561ae3b | | image | centos-7-slurm-20160422210744 (7364ada7-263e-4fb0-a9f4-219ab19e0be0) | | key_name | jhoblitt-slurm | | name | slurm-slave4 | | os-extended-volumes:volumes_attached | [] | | project_id | 8c1ba1e0b84d486fbe7a665c30030113 | | properties | slurm_node_type='slave' | | status | ERROR | | updated | 2016-05-02T20:29:57Z | | user_id | 83bf259d1f0c4f458e03f9002f9b4008 | +--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+ {code} ",2 DM-5928,"April 2016 LAAIM work","Drafted documentation for Web SSO capabilities: https://confluence.lsstcorp.org/display/LAAIM/Web+SSO Began testing new NCSA IAM capabilities (group management, user self-registration). Registered NCSA with InCommon as a sub-org of UIUC to ease future IdP/SP registrations. Attended local NCSA LSST coordination meetings.",4 DM-5929,"April Work for ConOps","Work on developing, editing and providing feedback for various ConOps. Converted existing ConOps to new format.",33 DM-5930,"Replace exiting DipoleMeasurementTask with DipoleFitTask","The goal of this ticket is to replace the existing DipoleMeasurementTask with the new DipoleFitTask, subsequent to ticket DM-5413. TBD: does this include completely removing all remnants of DipoleMeasurementTask code?",6 DM-5931,"Test planned implementation of Lupton(ZOGY) algorithm in real space","Develop a (1-D?) simple toy model and test the effects of the correction for varying I1 and I2 noise levels and different image PSFs and matching kernel(s). This will be done first in an ipython notebook. See DM-5914.",6 DM-5932,"Trial implementation of Lupton(ZOGY) in stack","The Lupton reinterpretation of the ZOGY algorithm in real-space is essentially a post-convolution that implements noise whitening (or decorrelation) of the image difference. We will make a first-pass at implementing this in the existing diffim codebase in order to perform future evaluations on real data.",16 DM-5933,"Replace jointcal.StarSelector with meas_algorithms.starSelector","jointcal has its own custom star selector. This should be removed and replaced with a star selector based on meas_algorithms.starSelector. A good choice might be meas_algorithms.objectSizeStarSelector.",2 DM-5934,"Update developer guide with Astropy guidance","Once RFC-178 is adopted the developer guide has to be updated to include guidance as to how Astropy can be used in the stack (similar to how Boost is documented).",1 DM-5935,"Package Astropy for the stack","Once RFC-178 is adopted Astropy needs to be packaged in an EUPS container. Given the complexity of Astropy dependencies the packaging will be done as for {{numpy}} and {{scipy}} by checking that Astropy is available (v1.1 will be the minimum version).",1 DM-5936,"Make afw rgb unit test PEP440 compliant for matplotlib check","If a user has a version of matplotlib installed from a git clone, the afw rgb unit test fails at the matplotlib version check. The versioning scheme for this type of install is determined by pep 440. Make the unit test properly handle this type of version comparison.",1 DM-5937,"April work for middleware","Participated in requirements definition",1 DM-5938,"Redirect non HTTPS requests on LSST the Docs to TLS","See https://docs.fastly.com/guides/securing-communications/allowing-only-tls-connections-to-your-site",1 DM-5939,"Pre-release versions of matplotlib 2.0 break afw unit tests","In the afw rgb unit test, testWriteStarsLegacyAPI checks to make sure that a file name with an unknown extension raises a value error. In current version of matplotlib, saving a file with an unknown extension causes this error: {code} *** ValueError: Format ""unknown"" is not supported. Supported formats: eps, jpeg, jpg, pdf, pgf, png, ps, raw, rgba, svg, svgz, tif, tiff. {code} In matplotlib 2.0 prerelease the file is saved as a png when an unknown extension is specified. Since the write call success the unit test fails as it is expecting a failure. If nothing depends on this behavior, the unit test should probably be removed.",1 DM-5940,"Create new build based on the converted firefly code.","- remove all of the gwt code except for a few remaining files. - create separate build for the new firefly viewer, leaving the old fftools as it was before the JS conversion. - repackage files as needed moving forward.",8 DM-5941,"Private network not available across all instances","I'm setting up an ELK system. Part of that is an Elasticsearch system. When I bring up the system the private network is bisected. I attempted creating a security group, in case that was a problem but it didn't help. Note that the work around is to create security groups or use a firewall and use floating ips. This is far from ideal. I think the right solution is to use the private network. Example: First section {{p-es-1}} {{p-es-3}} {{p-es-k}} {code:bash} vagrant@es-1:~$ ifconfig ens3 Link encap:Ethernet HWaddr fa:16:3e:47:28:a7 inet addr:10.0.42.30 Bcast:10.0.42.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe47:28a7/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1454 Metric:1 RX packets:363265 errors:0 dropped:0 overruns:0 frame:0 TX packets:304215 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:95396177 (95.3 MB) TX bytes:238466304 (238.4 MB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:850 errors:0 dropped:0 overruns:0 frame:0 TX packets:850 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1 RX bytes:138411 (138.4 KB) TX bytes:138411 (138.4 KB) vagrant@es-1:~$ ping 10.0.42.32 PING 10.0.42.32 (10.0.42.32) 56(84) bytes of data. 64 bytes from 10.0.42.32: icmp_seq=1 ttl=64 time=0.284 ms 64 bytes from 10.0.42.32: icmp_seq=2 ttl=64 time=0.266 ms 64 bytes from 10.0.42.32: icmp_seq=3 ttl=64 time=0.265 ms 64 bytes from 10.0.42.32: icmp_seq=4 ttl=64 time=0.302 ms ^C --- 10.0.42.32 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 2998ms rtt min/avg/max/mdev = 0.265/0.279/0.302/0.019 ms vagrant@es-1:~$ ping 10.0.42.34 PING 10.0.42.34 (10.0.42.34) 56(84) bytes of data. 64 bytes from 10.0.42.34: icmp_seq=1 ttl=64 time=0.333 ms 64 bytes from 10.0.42.34: icmp_seq=2 ttl=64 time=0.325 ms 64 bytes from 10.0.42.34: icmp_seq=3 ttl=64 time=0.322 ms 64 bytes from 10.0.42.34: icmp_seq=4 ttl=64 time=0.319 ms ^C --- 10.0.42.34 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 2998ms rtt min/avg/max/mdev = 0.319/0.324/0.333/0.022 ms vagrant@es-1:~$ ping 10.0.42.31 PING 10.0.42.31 (10.0.42.31) 56(84) bytes of data. From 10.0.42.30 icmp_seq=1 Destination Host Unreachable From 10.0.42.30 icmp_seq=2 Destination Host Unreachable From 10.0.42.30 icmp_seq=3 Destination Host Unreachable ^C --- 10.0.42.31 ping statistics --- 4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3017ms pipe 3 vagrant@es-1:~$ ping 10.0.42.33 PING 10.0.42.33 (10.0.42.33) 56(84) bytes of data. From 10.0.42.30 icmp_seq=1 Destination Host Unreachable From 10.0.42.30 icmp_seq=2 Destination Host Unreachable From 10.0.42.30 icmp_seq=3 Destination Host Unreachable ^C --- 10.0.42.33 ping statistics --- 4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3008ms pipe 3 vagrant@es-1:~$ ping 10.0.42.35 PING 10.0.42.35 (10.0.42.35) 56(84) bytes of data. From 10.0.42.30 icmp_seq=1 Destination Host Unreachable From 10.0.42.30 icmp_seq=2 Destination Host Unreachable From 10.0.42.30 icmp_seq=3 Destination Host Unreachable ^C --- 10.0.42.35 ping statistics --- 5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3999ms pipe 3 {code} Second section {{p-es-2}} {{p-es-4}} {{p-lfr}} {code:bash} vagrant@es-2:~$ ifconfig ens3 Link encap:Ethernet HWaddr fa:16:3e:6f:30:2c inet addr:10.0.42.31 Bcast:10.0.42.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe6f:302c/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1454 Metric:1 RX packets:196344 errors:0 dropped:0 overruns:0 frame:0 TX packets:160561 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:47667399 (47.6 MB) TX bytes:7135345 (7.1 MB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:97268 errors:0 dropped:0 overruns:0 frame:0 TX packets:97268 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1 RX bytes:8558096 (8.5 MB) TX bytes:8558096 (8.5 MB) vagrant@es-2:~$ ping 10.0.42.33 PING 10.0.42.33 (10.0.42.33) 56(84) bytes of data. 64 bytes from 10.0.42.33: icmp_seq=1 ttl=64 time=0.311 ms 64 bytes from 10.0.42.33: icmp_seq=2 ttl=64 time=0.309 ms 64 bytes from 10.0.42.33: icmp_seq=3 ttl=64 time=0.300 ms ^C --- 10.0.42.33 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.300/0.306/0.311/0.020 ms vagrant@es-2:~$ ping 10.0.42.30 PING 10.0.42.30 (10.0.42.30) 56(84) bytes of data. From 10.0.42.31 icmp_seq=1 Destination Host Unreachable From 10.0.42.31 icmp_seq=2 Destination Host Unreachable From 10.0.42.31 icmp_seq=3 Destination Host Unreachable From 10.0.42.31 icmp_seq=4 Destination Host Unreachable ^C --- 10.0.42.30 ping statistics --- 5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4014ms pipe 3 vagrant@es-2:~$ ping 10.0.42.32 PING 10.0.42.32 (10.0.42.32) 56(84) bytes of data. From 10.0.42.31 icmp_seq=1 Destination Host Unreachable From 10.0.42.31 icmp_seq=2 Destination Host Unreachable From 10.0.42.31 icmp_seq=3 Destination Host Unreachable From 10.0.42.31 icmp_seq=4 Destination Host Unreachable From 10.0.42.31 icmp_seq=5 Destination Host Unreachable ^C --- 10.0.42.32 ping statistics --- 5 packets transmitted, 0 received, +5 errors, 100% packet loss, time 4023ms pipe 3 vagrant@es-2:~$ ping 10.0.42.34 PING 10.0.42.34 (10.0.42.34) 56(84) bytes of data. From 10.0.42.31 icmp_seq=1 Destination Host Unreachable From 10.0.42.31 icmp_seq=2 Destination Host Unreachable From 10.0.42.31 icmp_seq=3 Destination Host Unreachable ^C --- 10.0.42.34 ping statistics --- 4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3006ms pipe 3 vagrant@es-2:~$ ping 10.0.42.35 PING 10.0.42.35 (10.0.42.35) 56(84) bytes of data. 64 bytes from 10.0.42.35: icmp_seq=1 ttl=64 time=0.387 ms 64 bytes from 10.0.42.35: icmp_seq=2 ttl=64 time=0.278 ms 64 bytes from 10.0.42.35: icmp_seq=3 ttl=64 time=0.288 ms ^C --- 10.0.42.35 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1998ms rtt min/avg/max/mdev = 0.278/0.317/0.387/0.053 ms {code} This can be reproduced by sourcing your OpenStack credentials and running this [{{Vagrantfile}}|https://gist.github.com/jmatt/7b6eb6a042c4e63531d40d1a68069f33]. Use {{vagrant ssh p-es-1}} to connect to the {{p-es-1}} instance. ",2 DM-5942,"Public elasticsearch configuration using SSL/TLS and basic auth","Create a (relatively) secure way to use Elasticsearch from outside of Nebula using Ansible. See: https://www.elastic.co/blog/playing-http-tricks-nginx Note that this will not allow another Elasticsearch to join but will allow the usual admin and client queries using basic auth.",1 DM-5943,"Add Git refs to Jobs Table of QA Dashboard","The Level 0 QA DB should know what Stack Git refs correspond to each job. This will enable plots to filter jobs based on development ticket so that a developer can understand how a branch compares to master. This ticket will add jobs to the schema (http://sqr-009.lsst.io/en/latest/#level-0-qa) and create the necessary migration script.",5 DM-5945,"Implement validate_drp plot in Bokeh as proof-of-concept for QA Dashboard","This ticket will implement a plot from validate_drp in the QA Dashboard as a proof-of-concept for how existing matplotlib plots can be re-implemented in Bokeh with data from the QA database. Stretch goals (maybe for a future ticket) will be to overplot the validate_drp output of one job against another’s to understand performance changes.",2 DM-5946,"SUIT design ","Finish SUIT design, produce a design document",75 DM-5947,"Jupyter widget using Firefly","Jupyter widget using Firefly visualization components. To better support the user community in using Jupyter notebook with Python packages, we want to start exploring the process of creating Jupyter widget, using Firefly visualization capabilities. ",26 DM-5948,"Workspace design","workspace design * ",30 DM-5949,"Testing framework setup and more unit test","Testing framework setup and more unit test code ",45 DM-5951,"Make obs_subaru PEP8 (pyflakes) compliment","Running pyflakes on obs_subaru revels many places where the code is not (LSST specific) PEP8 compliment. Actual coding bugs reviled by pyflakes were fixed in DM-5474, however many of the formatting issues need to be fixed. Once DM-4740 and DM-4668 are done, all remaining code should be brought into coding standard compliance.",3 DM-5952,"Initial draft(s) of the Data Backbone ConOps","Initial drafts of the data backbone concept of operations. Versions 0.1, 0.2. Produced document that is ready for friendly, internal review although document is not complete.",8 DM-5953,"Internal review of Data Backbone ConOps and revision(s)",NULL,2 DM-5954,"Watch Boot Camp materials","Videos from the [DM Boot Camp|https://community.lsst.org/t/dm-boot-camp-announcement/249] cover a lot of topics a newbie like me is interested in.",4 DM-5955,"Build LSST Software Stack from the source","Create a virtual machine with functioning LSST Software Stack to have an environment where I can see and play with its code.",3 DM-5958,"Integration Environment: top level design","Working with SUI/QServ teams to create a plan that includes deployment schedules, levels of support, discussions of administrative requirements in integration environment, detailed documentation before procurement, security reviews, reviews with deployment team.",26 DM-5959,"Integration Environment: Procurement","Discussions with vendors. Quote selection. Budget tracking. Quote submission to finance. GCO follow up questions. OBFS follow up questions. Finance follow up questions. Overall tracking of purchase progression.",10 DM-5960,"Node delivery to racking","Node delivery acceptance, unpacking, inventory, rack builds, pdu placement, node racking, bios updates, power connections, ",16 DM-5961,Networking,"Unpacking networking equipment, inventory, ordering/procure/unpack cabling, equipment racking, cabling, config creation and management.",6 DM-5962,"OpenStack deployment","This activity is outside of LSST project; this story is a placeholder for demonstrating progression toward completion of the epic. Some activities that are likely to occur: Software installation, OS installation/updates, RAID configurations, software configuration, monitoring and notification system(s) installation and configuration, integration testing and friendly uses evaluation. ",24 DM-5963,"Compute Upgrade","Discussions and planning for the creation of an LSST service-only tenant for better isolation (hence protection) from ad hoc services. Further discussions with LSST DM interested stakeholders about this feature. Policy development for use and monitoring of use of this feature.",10 DM-5964,"Object Storage Installation","Discussions and planning for allocating the 1PB storage increase within Nebula between object storage and block storage. Further discussions with LSST DM interested stakeholders about this feature. Policy development for use and monitoring of use of this feature.",5 DM-5966,"Remove use of Boost smart pointers in meas extensions","Removal of boost smart pointers in DM-5879 missed some meas extensions which are not built as part of {{lsst_distrib}}. Namely: {{meas_extensions_shapeHSM}}, {{meas_extensions_simpleShape}} and {{meas_extensions_photometryKron}}. Update these too.",1 DM-5967,"Provide docker swarm POC for Qserv containers orchestration",NULL,10 DM-5968,"Split secondary index loading from qserv_data_loader.py to separate unit test","To isolate development and validation of secondary index loading strategies, encapsulate loading of ""pure"" secondary index data via qserv_data_loader.py, without using entire datasets.",8 DM-5969,"Deploy any secondary index loader mods into qserv_data_loader.py","Following completion of DM-5968, any modifications to the top-level data loading procedure for the secondary index need to be deployed back to the main qserv_data_loader.py driver.",5 DM-5970,"Slurm deployment preparation","Installation and familiarization with the supported level of service of slurm installation. Includes rolling that configuration into puppet modules/manifests. Also includes documentation of the installation and configuration requirements. This is month-of-April work; a new story continues into May.",20 DM-5972,"Inventory to racking","Inventory, unpacking, bios updating, racking, rack building, pdu installation, power cabling, OS installation, RAID configuration, xCAT deployment planning, Puppet deployment planning, puppet module/manifest creation, puppet config versioning, cabling strategies discussions, additional cable purchases (minor).",30 DM-5973,"Update developer guide with pytest guidance","Now that DM-5561 explains how to migrate to pytest compatibility the developer guide must be updated to state how to use pytest in unittests.",5 DM-5976,"Change SubtractBackgroundConfig.isNanSafe default to True","[~price] suggests that the default value for {{SubtractBackgroundConfig.isNanSafe}} be changed from False to True.",1 DM-5977,"Create and deploy Beats for Logstash and Elasticsearch.","Create and deploy Beats for Logstash and Elasticsearch. These beats are used to transport logs and monitoring data to ELK. See: https://www.elastic.co/products/beats",3 DM-5978,"Miscellaneous Nebula service items for x16","This story is for miscellaneous Nebula service items that do not have individual LSSTDM JIRA issues (often handled in the RT ticket system) including account creation requests, reporting on hanging/errant processes for cleanup, response & communiques on security incidents, etc. ",8 DM-5979," tests in testArgumentParser.py fail Jenkins run-rebuild on nfs","(1) {{testOutputs}} fails because paths are compared literally Jenkins run-rebuild #139 failed with pipe_base https://ci.lsst.codes/job/run-rebuild/139//console {code:java} FAIL: testOutputs (__main__.ArgumentParserTestCase) Test output directories, specified in different ways ---------------------------------------------------------------------- Traceback (most recent call last): File ""tests/testArgumentParser.py"", line 497, in testOutputs self.assertEqual(args.input, DataPath) AssertionError: '/nfs/home/lsstsw/stack/Linux64/obs_test/2016_01.0-3-gafa6dd0+10/data/input' != '/home/lsstsw/stack/Linux64/obs_test/2016_01.0-3-gafa6dd0+10/data/input' {code} Please make the comparison more robust. (2) File descriptor leaks Jenkins run-rebuild #138 failed with pipe_base https://ci.lsst.codes/job/run-rebuild/138//console {code:java} FAIL: testFileDescriptorLeaks (lsst.utils.tests.MemoryTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File ""/home/lsstsw/stack/Linux64/utils/2016_01.0-2-g97a6e33/python/lsst/utils/tests.py"", line 133, in testFileDescriptorLeaks self.fail(""Failed to close %d files"" % len(diff)) AssertionError: Failed to close 1 files {code} {code:java} File open: /nfs/home/lsstsw/build/pipe_base/.nfs000000000a20a3f700005679 {code} This test passes on local disk. ",2 DM-5980,"update jenkins to 2.x",NULL,10 DM-5983,"Stop cleanly MySQL if configuration step fails","Next scripts doesn't stop cleanly MySQL if configuration step fails: {code:bash} admin/templates/configuration/tmp/configure/scisql.sh admin/templates/configuration/tmp/configure/tools/sql-loader.sh {code}",2 DM-5984,"Use RO MySQL account in qserv_testdata","It seems integration tests datasets are now loaded with Loader. So using MySQL root account is no more required in integration tests, an account with SELECT access on test databases (like qsmaster), should be enough. Furthermore, all code related to MySQL writes can be removed from {code:bash} python/lsst/qserv/tests/sql/cmd.py python/lsst/qserv/tests/sql/connection.py {code}",4 DM-5985,"Add unicode support for Qserv password","Qserv password must be encoded in ASCII for now in qserv-meta.conf. Unicode passwords should be supported.",6 DM-5986,"Use sagas in place of side-effects in chart-related controllers ","Replace side-effects with saga and clean-up chart related controllers (TableStats, XYPlot and Histogram).",3 DM-5988,"Support Monocam reduction","Monocam is being used on a telescope, and we want to reduce the data obtained. This is made difficult by the fact that the camera and the telescope are not talking to each other so the usual header keywords are in separate files from the data.",6 DM-5991,"A look at the overall performance of the application","Investigate the overall performance of the application and improve it where possible. It is pointed out that triview is especially slow compare to expanded. Need to investigate.",4 DM-5992,"Reception and Placement","Receive, unbox, inventory, inspect, build racks, rack nodes and power.",10 DM-5993,"Networking Configuration","Unbox, inspect, rack, power networking equipment, cables ordered, alternate cable purchase options tested, initial switch configuration(s), cabliing.",15 DM-5994,Provisioning,"OS + updates installation, imaging building, stateless/stateful node provisioning, test of image, security configurations, networking bandwidth tuning, file system tuning, file system installation and tuning.",39 DM-5995,"Disaster Recovery Implementation","Testing/practicing recovery of node/image/software after various types of faults.",30 DM-5996,Documentation,"Document each component sufficient enough for transfer of knowledge and system recovery as needed.",5 DM-5997,"Capability Validation","Review that design was implemented successfully including recover and supporting documentation exists.",10 DM-5998,"Security Vetting","Review of capability by site security team",3 DM-5999,"Acceptance by Stakeholders","Review with stakeholders (target users, release manager, others as necessary) to confirm that capability fulfills original requirements. ",2 DM-6004,"Acceptance by Stakeholders","Review of services (compute, storage, networking) by LSST project before considering work final.",5 DM-6005,Procurement,NULL,8 DM-6006,"Reception and Placement",NULL,3 DM-6007,"Networking Configuration",NULL,10 DM-6008,Provisioning,NULL,30 DM-6009,"Disaster Recovery Implementation",NULL,20 DM-6010,Documentation,NULL,15 DM-6011,"Capability Validation ",NULL,15 DM-6012,"Security Vetting",NULL,10 DM-6013,"Acceptance by Stakeholders",NULL,20 DM-6014,"Reception and Placement",NULL,2 DM-6015,"Networking Configuration",NULL,3 DM-6016,Provisioning,NULL,10 DM-6017,"Disaster Recovery Implementation",NULL,4 DM-6018,Documentation,NULL,4 DM-6019,"Capability Validation",NULL,5 DM-6020,"Security Vetting",NULL,6 DM-6021,"Acceptance by Stakeholders",NULL,6 DM-6022,"Lazy load related chart data on table data update","When new table data received, the related chart data should be updated only for the components on display. Hidden components' data should be lazily updated when a component becomes visible.",3 DM-6025,"ingest.py throwing away errors","Line 118 of ingest.py has a problem try block which is currently just throwing away errors, which has made for some confusing/frustrating debugging. This should be changed to either warn or raise, but not silently dispose of errors.",1 DM-6026,"Make it possible to distinguish TABLE_NEW_LOADED actions triggered by sort","It would be beneficial to have in the TABLE_NEW_LOADED payload a trigger field, which would differentiate actions triggered by sort (where data do not change, only their order) or filter from other loads. We don't need to reload table statistics or histogram on sort. But we do need to to reload them on filter. created TABLE_SORT action to distinguish sorting from filtering. sorting should not reload xyplot nor catalog overlay. Also: - disable history when in api mode. - ensure tableMeta.source reflects the file on the server. - fix TablePanelOptions not resetting columns selection. - remove 'Fits Data' tab when no images available. - fix 'Coverage' appearing when it should.",2 DM-6028,"Validation is not performed on unchanged fields","Currently, validation is performed only if a field has changed. We need to be able to validate all fields on form submit. The issue is not limited to initial (ex. empty) value being invalid. The invalid message is lost when a field is unmounted/re-mounted. You can test the following way: - http://localhost:8080/firefly/;a=layout.showDropDown?view=AnyDataSetSearch - Open chart settings, enter 1000 into X/Y ratio - the field is shown as invalid - Switch to histogram and back, the invalid message is gone, the field appears to be valid Another test case is Example Dialog tab 'X 3', 'X 3' tab test field initial value 88 is invalid (it should be between 22 and 23), but it appears valid. ",2 DM-6029,"Error message is not shown","The error message is not showing consistently when mouse is over tha exclamation icon.",1 DM-6030,"Investigate possibilty of cosmic ray muons (etc) for precision gain calibration","In the era of CBPs, we care about absolute system throughput, and thus need to accurately know the gain of amplifiers in the CCDs. Initially, this can be done by lab-based Fe55 characterisation (modulo the non-linearity, though that itself will need to be need to be characterised and corrected for), but changes in the relative gains of the various amplifiers need to be monitored, and this must be done in a way that is not degenerate with the optical transmission in any way. Theoretically it should be possible to use cosmic ray muon tracks, and tracks from radioisotope contamination of the glass/dewar, to measure the (change in the) relative gains of the amplifiers. Early work has shown that this does work in principle, but this ticket is for some further effort to see whether this method can provide the necessary accuracy given the amount of data available remains to be seen. This ticket would normally need to be significantly more points, but as it builds on earlier work, it can, at least for initial results, be done quite cheaply. Initial investigation will inform further work. 1st order: histogram all pixels in dark images after careful bias subtraction. Look at shape of spectrum. Fit some arbitrary function, and correlate with Fe55 gain measurements. 2rd order: Separate muon tracks from soft electron tracks/nuclear recoil events. Can cutting one of more of these types out improve the correlation? Or can treating them separately improve the resolution? 3rd order: Recalculate the histogram in terms of the dE/dx for the muon tracks, i.e. taking into account their track lengths and thus angle through the silicon.",10 DM-6031,"Create documentation for bright object masks","The bright object mask code ported from hsc bought the ability to mask regions, during coaddition, by providing mask files. How to create these files, and where they should be placed in the file system is documented on and HSC ticket (https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1351) but not on the LSST side. The information on how to create and use bright object masks should be put into LSST documentation.",3 DM-6033,"Support sphinxcontrib-bibtex in technotes","Allow bibtex-based references in technotes using [sphinxcontrib-bibtex|https://github.com/mcmtroffaes/sphinxcontrib-bibtex]. DMTN-010 will be used as a pilot case.",1 DM-6036,"Produce and ingest master calibs for USNO monocam data.","Use the construct*.py scripts added to pipe_drivers to produce temporally relevant master biases, darks, flats (and fringe frames?) for the recent USNO observing with monocam. A small amount of hacking will be required due to the fact that the current ingestion model assumes that each CCD frame has a USNO counterpart which tells about the telescope pointing etc, but the bias frames do not have these. Once the master calibs are produced, get them ingested.",2 DM-6037,"Reduce sky data from USNO monocam run","Using the master calibs produced in DM-6036, push all the monocam data through processCcd. Others will run sanity checks on the output (initial astrometry & photometry). From there I believe people will look at using the data to test jointcal & sim_astrom etc, but this ticket just related to the initial reduction. As more data comes in from the 2nd telescope, a little further hacking may be necessary to keep everything running. Some of this will likely be hacky or need one-off solutions/header modification, hence the higher-than-normal number of story points assigned to what one might expect to be an hour-long job.",3 DM-6038,"Monocam bias structure analysis","Estimates of the noise in the bias frames coming from the USNO monocam run around ~20e- RMS. This is higher than with the same readout configuration in the lab, and could be due to several things. This ticket is to take a look at a few bias frames and investigate the structure of the noise. If it is periodic and at a constant phase then using master biases will significantly improve the SNR in the calexps produced, but if it is not, then whether or not bias frames should be used at all should be considered.",1 DM-6039,"Download temporally relevant raw calibs for CTIO DECam data","Go to the NOAO portal and download a sufficient number of darks, biases and flats (in each band) from around the time of the CTIO trip to produce master calibs.",1 DM-6040,"Create and ingest master calibs for DECam CBP reduction","Having collected data in DM-6039, push all this through the master calib creation scripts and ingest master calibs into registry. This is a necessary but not-necessarily-sufficient ingredient for making progress on DM-5465.",2 DM-6041,"Functional use cases for tools","Work with Vandana Desai (IRSA) to tabulate science use cases for tools. Then transform the science use cases to functional use cases (""this is how we want the tool/interface to behave""). This work is to identify common functional and scientific use cases between PTF/ZTF and LSST to inform LSST on how the SUIT web portal might be organized for user interaction with LSST data. ",1 DM-6042,"Add viewer launching API","Add ability to launch the viewer and load images, xyplots, and tables from the api. We use to call this firefly.getExternalViewer() Also, we have this concept of 'root path' through out the code. The api use can set a root path so he can use his when we are cross site. Need to implement. ",6 DM-6043,"Change mouse readout to use supports MouseReadoutCntlr & add an API readout","Change mouse readout to use MouseReadoutCntlr. * Use {{VerySimpleMouseReadout}} as a reference. * Change to use {{MouseReadoutCntlr}} for options instread of ImagePlotCnltr * Add a second (vertical) Mouse readout to be used in the api mode. The readout should be set into {{ApiUtilImage.jsx}}, {{initAutoReadout}}. It should replace {{VerySimpleMouseReadout}} & add an API readout",10 DM-6048,"Bundle up more HSC data for validate_drp","We would like to include a larger set of HSC data for validation. I tested this while in Tucson. My working dir was {{/tigress/pprice/frossie}}. The raw and processed data should be stuffed into validation_data_hsc",1 DM-6050,"Table caching optimizations","We need to avoid duplicate requests which result from minor differences in TableRequest parameters, which are not used to get data. For example, loading catalog table, which triggers table statistics, and then getting an XY plot, results in 3 requests, returning identical data. 1. RequestClass=ServerRequest; *tbl_id=tbl_id-1;* UserTargetWorldPt=10.68479;41.26906;EQ_J2000;m31;ned; SearchMethod=Cone; catalog=wise_allwise_p3as_psd; RequestedDataSet=wise_allwise_p3as_psd; radius=200; use=catalog_overlay; catalogProject=WISE 2. RequestClass=ServerRequest;RequestedDataSet=wise_allwise_p3as_psd; catalog=wise_allwise_p3as_psd; use=catalog_overlay; UserTargetWorldPt=10.68479;41.26906;EQ_J2000;m31;ned; catalogProject=WISE; radius=200; SearchMethod=Cone 3. RequestClass=ServerRequest; *tbl_id=xyplot-tbl_id-1;* catalog=wise_allwise_p3as_psd; use=catalog_overlay; UserTargetWorldPt=10.68479;41.26906;EQ_J2000;m31;ned; SearchMethod=Cone; RequestedDataSet=wise_allwise_p3as_psd; catalogProject=WISE; radius=200; *decimate=decimate=ra,dec,10000,1,,,,* The difference between 1 and 2 is tbl_id parameter. The difference between 2 and 3 is tbl_id and decimate parameters. As well as the order of the parameters. None of which change the catalog search result. Test Case: Test Searches, Test catalog, AllWISE Source, radius=200",2 DM-6051,"Add extendedness vs. star selector test to single-visit validation in ci_hsc","ci_hsc has a test that verifies that extendedness as measured on coadds broadly agrees with the star selection done for PSF estimation on individual frames. This tests a bunch of stuff, including aperture corrections on the coadds and propagation of flags from visits to coadds. It doesn't test that aperture correction vs. extendedness logic is correct in processCcd.py, but just copying this test to the appropriate validation function in ci_hsc should do the trick. This is currently broken, but should be fixed in DM-5877.",2 DM-6052,"Improve password management for Qserv MySQL accounts","Qserv passwords management for MySQL account (i.e. root, monitor, qsmaster) should be improved. See wmgr password management to have a good example. Furthermore qsmaster use currently empty password, this must be fixed.",8 DM-6053,"Allow use of other MySQL account thant 'qsmaster'","'qsmaster' value can't be changed in qserv-meta.conf, this must be fixed {code} diff --git a/admin/templates/installation/qserv-meta.conf b/admin/templates/installation/qserv-meta.conf index 81b6ca9..203ebda 100644 --- a/admin/templates/installation/qserv-meta.conf +++ b/admin/templates/installation/qserv-meta.conf @@ -103,7 +103,7 @@ user_monitor = monitor password_monitor = CHANGEMETOO # Used to access Qserv data and metadata (like indexes) -user_qserv = qsmaster +user_qserv = qservdata {code} Above change leads to next error in integration tests: {code:bash} 154 [0x7f1beacf9700] ERROR lsst.qserv.sql.SqlConnection null - connectToDb failed to connect! 154 [0x7f1beacf9700] ERROR lsst.qserv.sql.SqlConnection null - runQuery failed connectToDb: START TRANSACTION 2016-05-10 14:55:09,684 - root - CRITICAL - Exception occured: Error from mysql: (-999) Error connecting to mysql with config:[host=127.0.0.1, port=13306, user=qsmaster, password=XXXXXX, db=qservCssData, socket=] Traceback (most recent call last): File ""/home/dev/src/qserv/bin/qserv-data-loader.py"", line 274, in <module> loader = Loader() File ""/home/dev/src/qserv/bin/qserv-data-loader.py"", line 225, in __init__ css_inst = css.CssAccess.createFromConfig(config, """") CssError: Error from mysql: (-999) Error connecting to mysql with config:[host=127.0.0.1, port=13306, user=qsmaster, password=XXXXXX, db=qservCssData, socket=] 2016-05-10 14:55:09,810 - lsst.qserv.admin.commons - CRITICAL - Error code returned by command : qserv-data-loader.py -v --config=/qserv/stack/Linux64/qserv_testdata/2016_01-1-g7b10791+7/datasets/case05/data/common.cfg --host=127.0.0.1 --port=5012 --secret=/home/dev/qserv-run/git/etc/wmgr.secret --delete-tables --config=/qserv/stack/Linux64/qserv_testdata/2016_01-1-g7b10791+7/datasets/case05/data/Object.cfg --css-remove --skip-partition --chunks-dir=/home/dev/qserv-run/git/tmp/qservTest_case05/chunks/Object --config=/qserv/stack/Linux64/qserv_testdata/2016_01-1-g7b10791+7/datasets/case05/data/Object.cfg --empty-chunks=/home/dev/qserv-run/git/var/lib/qserv/empty_qservTest_case05_qserv.txt qservTest_case05_qserv Object /qserv/stack/Linux64/qserv_testdata/2016_01-1-g7b10791+7/datasets/case05/data/Object.sql /home/dev/qserv-run/git/tmp/qservTest_case05/chunks/Object/Object.txt ERROR {code} ",4 DM-6054,"Minor updates in suptertask from following DMTN-002","Some examples in the DMTN-002 seem slightly out of date. Update supertask documentation and code to catch up with some recent developments in the stack. ",2 DM-6057,"v12.0 [Winter 2016 / Extra 2016] release","This is the ticket for the v12.0 release prep. Edit: Release announcement at https://community.lsst.org/t/lsst-stack-version-12-0-winter-2016-extra-2016-release/874 ",12 DM-6062,"Launch integration tests using Docker+Openstack","vagrant and packer will be replaced by openstack python API and cloud-init which are more flexible. - A pseudo-DNS will be provided thanks to avahi/mdns - Only one public/floating IP willl be used (in order to allow booting large clusters laters)",10 DM-6063,"Fix how aperture correction is applied","[~lauren] committed a fix Jan 15 to how aperture correction is applied that I accidentally lost when refactoring in DM-4692. https://github.com/lsst/pipe_tasks/commit/d904e3d188698b4f57bf3dad1516b0bf201078f5 Restore the fix. The need for this fix suggests a design flaw in measurement that will be fixed as part of DM-5877",1 DM-6064,"Define, design, and RFC repository refactor.","Includes support for * butler manages input & output repos * repo tagging ",18 DM-6065,"design work for butler storage & format factorization",NULL,5 DM-6069,"Camera team visualization support (F16)","Support the camera team to use Firefly for their visualization needs.",12 DM-6071,"L1 Messaging path status","All principle entities for L1 are in place and the messaging is working as intended. The message dictionary includes some message types for prototyping and will likely double in size as the interface between OCS is firmed up in the coming weeks. The implementation thus far is a 'ready, set, go' set of states. Test files from 10 Forwarders are sent to 10 distributors thru a WAN emulation device, and the result can be timed. The DMCS component is a simple CLI to initiate messages as currently written. This component will be expanded as requirements are determined. Still under development is a component layer between the Condor controller and the NCSA Foreman entity so that resource availability can be queried and provide a communication link for ancillary information as needed.",40 DM-6072,"Li prototype code and the Wan Emulator","L1 Forwarder components and Distributor components are located on opposite sides of the Emulator (the Long Haul network component) and move files across the path when configured and given a 'go' signal via messaging. Forwarder/Distributor pairs are set up dynamically for each file transfer (similar to a readout event). Results are temporarily forwarded to a status queue sink, where messages are processed for the publishing of results.",30 DM-6073,"Pass background to NoiseReplacerTask","Implement RFC-180: `NoiseReplacerTask` wants some statistics about the background that was subtracted from the exposure, but it gets these in a fragile and roundabout fashion: it expects the code that measures the background to put the mean and variance into the exposure's metadata, using special keys. It is difficult to enforce correctness because background is measured several times while processing an exposure. To solve this, pass the background directly to `NoiseReplacerTask`. This will require passing the background through the various measurement tasks, which will require small changes to code that calls the measurement tasks. In addition, remove computation of background statistics from the background fitting code.",4 DM-6074,"Add RegistryField support to Task.makeSubtask","As part of implementing RFC-183 add support for tasks specified in {{lsst.pex.config.RegistryField}} to {{lsst.pipe.base.Task.makeSubtask}} ",2 DM-6075,"Document the need for abstract base tasks for tasks","As part of RFC-183 document the fact that variant tasks should have a common abstract base class that defines the API. If we add future tasks that we feel are likely to have variants, then we should create an abstract base class. Candidates include star selectors, PSF determiners and ISR tasks. Note that this applies to tasks LSST provides in its stack, not to variants users produce and other obscure one-off code. Also document the desire that tasks with anticipated many variants, such as star selectors, and PSF determiners should be in registries. This explicitly excludes tasks such as ISR where only one task is likely to be useful for a given set of data. ",2 DM-6076,"Create a registry for star selectors","Create a registry for star selectors and use the registry instead of ConfigurableField in tasks that call a star selector. Update config overrides in obs_* packages and unit tests accordingly.",3 DM-6077,"Change PSF determiners into tasks","PSF determiners are already configurables, and some benefit from having a log. Take the logical next step and make them instances of {{lsst.pipe.base.Task}}.",1 DM-6078,"Aperture correction fails to measure a correction for the final plugin in the list and reports misleading errors","Since the refactoring of DM-4692, runs of *processCcd.py* detail the following in their logs: {code:title=With base_PsfFlux and base_GaussianFlux plugins registered} processCcd.charImage.detectAndMeasure.measureApCorr WARNING: Only 0 sources for calculation of aperture correction for 'base_PsfFlux'; setting to 1.0 processCcd.charImage.detectAndMeasure.measurement: Measuring 65 sources (65 parents, 0 children) processCcd.charImage.detectAndMeasure.measurement.applyApCorr: Applying aperture corrections to 1 flux fields processCcd.charImage.detectAndMeasure.measurement.applyApCorr: Use naive flux sigma computation ... processCcd.calibrate.detectAndMeasure.measurement.applyApCorr: Applying aperture corrections to 2 flux fields processCcd.calibrate.detectAndMeasure.measurement.applyApCorr: Use naive flux sigma computation processCcd.calibrate.detectAndMeasure.measurement.applyApCorr WARNING: Could not find base_GaussianFlux_flux or base_GaussianFlux_fluxSigma in apCorrMap {code} {code:title=With base_PsfFlux, base_GaussianFlux, and ext_photometryKron_KronFlux plugins registered} processCcd.charImage.detectAndMeasure.measureApCorr: Measuring aperture corrections for 2 flux fields processCcd.charImage.detectAndMeasure.measureApCorr WARNING: Only 0 sources for calculation of aperture correction for 'base_PsfFlux'; setting to 1.0 processCcd.charImage.detectAndMeasure.measureApCorr WARNING: Only 0 sources for calculation of aperture correction for 'base_GaussianFlux'; setting to 1.0 processCcd.charImage.detectAndMeasure.measurement.applyApCorr: Applying aperture corrections to 2 flux fields processCcd.charImage.detectAndMeasure.measurement.applyApCorr: Use naive flux sigma computation ... processCcd.calibrate.detectAndMeasure.measurement.applyApCorr: Applying aperture corrections to 3 flux fields processCcd.calibrate.detectAndMeasure.measurement.applyApCorr: Use naive flux sigma computation processCcd.calibrate.detectAndMeasure.measurement.applyApCorr WARNING: Could not find ext_photometryKron_KronFlux_flux or ext_photometryKron_KronFlux_fluxSigma in apCorrMap {code} I can confirm that for the latter, running HSC data with the fix on DM-6063, the aperture corrections are being measured and applied for the PsfFlux and GaussianFlux measurements, but NOT for the KronFlux measurements. Looking at the output from the current ""expected"" values for the {{lsst_dm_stack_demo}} we see that there is an offset in the Psf-Gaussian fluxes, implying the Gaussian fluxes are not being measured (and hence not applied): !demo_current.png|width=500! From this I conclude that the aperture corrections are indeed being measured for all but the final entry in the plugin list. This implies that the report of ""Only 0 sources for calculation of aperture correction for 'xxx_xxFlux'; setting to 1.0"" is incorrect for all but the final plugin measurement. The demo previously successfully calculated aperture corrections and, after the logic fix of DM-4836, applied them in the correct order: !demo_previous.png|width=500! The sources of these issues and fixes for them are the goal of this issue.",2 DM-6079,"description of archive in a box","Please put in more detailed description of the ""Archive in a box"" concept",1 DM-6082,"Add Sublime Text configuration tips to Developer Documentation","[~rowen] and [~Parejkoj] have some good tips about setting up Sublime Text. [~jsick] suggested that we add these configuration tips to the Developer Documentation. http://developer.lsst.io/en/latest/#part-tools Both want to include info about recommended packages, but also the linter configurations to support the DM styles. I paste in here various helpful parts from the HipChat Software Development room discussion of this. Both verbatim, and summarized. 1. Install {{Package Control}} 2. Packages: {{Git}}, {{GitGutter}}, {{SideBarEnhancements}}, {{SublimeLinter}}, {{SublimeLinter-flake8}}, {{SublimeLinter-html-tidy}}, {{SumNumbers}}, {{Gist}}, {{BracketHighlighter}}, {{TrailingSpaces}}, {{Trimmer}}, {{OmniMarkupPreviewer}}, {{ReStructuredTextImproved}}, {{MarkDown Editing}}, {{Colorsublime}} 3. Themes: {{Sunburst}} color scheme * VIM users: {{Vintageous}} + Mac OS X configuration: {{defaults write com.sublimetext.3 ApplePressAndHoldEnabled -bool false}} so that holding down 'j' moves downward. Note that {{Vintageous}} is not a complete implementation of {{vim}}, but it at least allows enough basics so that one doesn't go crazy switching back and forth. link the {{subl}} command to {{/usr/local/bin}} Quick Tips: ""option-select (to select blocks) and select something then cmd-D are both extremely useful for modifying lots of things at once."" ""Similarly, ctrl-shift-up/down arrow."" ""cmd-click on multiple lines to have multiple synchronized cursors"" Configurations: 1. [~rowen]'s SublimeText Preferences file: https://jira.lsstcorp.org/secure/attachment/27846/Preferences.sublime-settings 2. Configuration {{flake8}} so that it works in the linting can take a bit of work if {{flake8}} isn't in your default path. See SublimeLinter.sublime-settings attachment for [~rowen]'s configuration: https://jira.lsstcorp.org/secure/attachment/27845/SublimeLinter.sublime-settings The above are useful, but we'll need someone to detail the linter stuff more.",1 DM-6083,"Enable websocket client to pickup channel parameter from url","send websocket channel information via url. keep channel information on browser reload. This is needed for Firefly Python API and external (when Firefly viewer is invoked trough URL) API. ",1 DM-6086,"JSON Schema for metric data from validate_drp to be ingested by the QA Dashboard app","A well-defined JSON schema is needed for {{validate_drp}}’s JSON output so that it can be easily, and consistently ingested into the QA Database. The schema will also make the JSON more self-describing, and potentially useful for other tools to build upon as well. The schema is being drafted in a thread at https://community.lsst.org/t/json-schema-for-squash/777?u=jsick. Once an informal consensus is reached the schema will be implemented in {{validate_drp}} on this ticket.",8 DM-6087,"jenkins job to execute validate_drp and push results to qa dashboard","This is the initial jenkins job that ""ties"" all the components together. It needs to: * execute validate_drp * push metadata about the jenkins build to qa dashboard * push the validate_drp metrics to qa dashboard",7 DM-6089,"Use fixed width integer types from std instead of boost","The following fixed width integer types are used in the stack: * {{boost::int16_t}} * {{boost::int32_t}} * {{boost::int64_t}} * {{boost::int8_t}} * {{boost::uint16_t}} * {{boost::uint32_t}} * {{boost::uint64_t}} * {{boost::uint8_t}} This ticket aims to replace them with their equivalents from {{cstdint}}.",1 DM-6098,"draw a diagram of DRP data flow","Study Jim Bosch's diagrams and descriptions (Parallelization in Data Release Production, Data Release Production Top-Level Overview), consider inputs/outputs of high level pipelines and parallelization of the DRP, draw a diagram to illustrate the data flow. ",4 DM-6099,"Improve afw.table Astropy view support","DM-5641 completed the first version of Astropy view support, but there is still room for improvement: - Make {{Footprint}} s in {{SourceCatalog}} s available as a {{dtype=object}} column. Same for {{Psf}} , {{Wcs}} , {{Calib}} in {{ExposureCatalog}}. - Use Astropy's coordinate classes for Coord fields (may require an RFC to determine how much we want to use Astropy's coordinate classes). ",4 DM-6100,"afw/tests/rgb.py fails due to .ttf files","afw/tests/rgb.py fails for me with the below error. We likely shouldn't be trying to track system resources like fonts, as we don't have any control over them. {code} [2016-05-12T19:46:12.528961Z] Failed test output: [2016-05-12T19:46:12.536029Z] tests/rgb.py [2016-05-12T19:46:12.536057Z] [2016-05-12T19:46:12.536070Z] ...s......ss...F. [2016-05-12T19:46:12.536106Z] ====================================================================== [2016-05-12T19:46:12.536138Z] FAIL: testFileDescriptorLeaks (lsst.utils.tests.MemoryTestCase) [2016-05-12T19:46:12.536173Z] ---------------------------------------------------------------------- [2016-05-12T19:46:12.536192Z] Traceback (most recent call last): [2016-05-12T19:46:12.536261Z] File ""/Users/parejkoj/lsst/lsstsw/stack/DarwinX86/utils/2016_01.0-4-g52f464f/python/lsst/utils/tests.py"", line 134, in testFileDescriptorLeaks [2016-05-12T19:46:12.536330Z] self.fail(""Failed to close %d file%s"" % (len(diff), ""s"" if len(diff) != 1 else """")) [2016-05-12T19:46:12.536352Z] AssertionError: Failed to close 2 files [2016-05-12T19:46:12.536356Z] [2016-05-12T19:46:12.536391Z] ---------------------------------------------------------------------- [2016-05-12T19:46:12.536404Z] Ran 17 tests in 3.451s [2016-05-12T19:46:12.536407Z] [2016-05-12T19:46:12.536424Z] FAILED (failures=1, skipped=3) [2016-05-12T19:46:12.536445Z] File open: /Library/Fonts/NISC18030.ttf [2016-05-12T19:46:12.536479Z] File open: /System/Library/Fonts/Apple Color Emoji.ttf [2016-05-12T19:46:12.536495Z] The following tests failed: [2016-05-12T19:46:12.539928Z] /Users/parejkoj/lsst/lsstsw/build/afw/tests/.tests/rgb.py.failed [2016-05-12T19:46:12.540060Z] 1 tests failed {code}",1 DM-6102,"implement basic oauth2 authentication for qa-dashboard","Per discussion at the SQRE co-working session on Thursday, we agreed to implement minimal authentication for the MVP version of the qa dashboard as an external reverse proxy, such as https://github.com/bitly/oauth2_proxy.",6 DM-6105,"Implement the post_save mechanism to update bokeh sessions when new data is available","In tickets/DM-5750 the bokeh python library was integrated in the squash django project. In order to exemplify its use the KPM CI chart is showing only hardcoded values for now. In this ticket we will implement methods to read the data from the database and the post_save mechanism to update the bokeh session when new data is available. ",4 DM-6106,"color map in visualization","the four new colormaps introduced in matplotlib last year http://bids.github.io/colormap/ d3js cmap: http://bl.ocks.org/mbostock/3289530 D3 supports CIELAB (Lab) and CIELCH (HCL) color spaces, which are designed for humans rather than computers. http://bl.ocks.org/mbostock/3014589 ",4 DM-6107,"Firefly performance profiling and code refactoring if needed","We need to dedicate some effort in each cycle to do the performance profiling and code refactoring needed to improve performance",22 DM-6108,"More work on firefly viewer layout control","More work needs to be done on the triview layout controlling: * When there is a table with image meta data is loaded, the images need to show with the meta data tab selected * When any data is pushed then drop downs needs to close * when a table a catalog table is loaded and there are no plots then then the tri-view should be up with the coverage tab selected. When there is plots then the coverage tab should not be selected. * we need a way to remove load a table and then only see the table, same with xy-plots * catalog and image meta data are determined by looking at the data. However, we might need this logic in a single function * when a table is loaded and we cannot determine what type it is then the table and the xyplots only should some up. * When all data is deleted the default tab should open. (in IRSAViewer case the is the select image panel)",8 DM-6111,"Browsers should cache editions for a shorter time period than Fastly","Currently we set {{Cache-Control: max-age=31536000}} so that Fastly caches uploads from LTD Mason for a year on its POPs. This has the side-effect of also having browsers potentially cache documentation on the client for up to a year. In practice, browsers churn through their cache space more quickly, but I've noticed that Safari has no cap on its cache space, and therefore can hold onto pages for a long time. The solution is to set a {{Surrogate-Control}} max age to 1 year, and have {{Cache-Control: max-age=0, private, must-revalidate}}. This will be done on LTD Keeper during the copy phase of a build into an edition (since it is reasonable for a client to cache a build forever), but then give us the flexibility to update an edition instantly. In the future we may want a more nuanced solution where CSS and JavaScript, for example, are cached longer on the browser.",1 DM-6112,"Provide minimal documentation for meas_extensions_photometryKron","Please provide a minimal level of documentation for meas_extensions_photometryKron, to include: * A doc directory with the usual content so that docstrings get generated by Doxygen; * A package overview; * All docstrings should be appropriate for parsing by Doxygen (ie, should start with {{""""""!}} where necessary). ",1 DM-6113,"Calibration Products Pipeline work during F17","This will need to be properly fleshed out before scheduling: for now, it's a bucket for stories that [~mfisherlevine] is unlikely to complete in X16. [~rhl], [~mfisherlevine] & [~swinbank] to provide further definition.",65 DM-6118,"The color stretch dialog box does not work properly","There are a few issues in the Color Stretch dialog box: # When the asinh or gamma algorithm is selected, the asinh parameters and gamma parameters are always reset to the default. The user-entered values can not be kept and used. # When there are two or more images, when the lower/upper range in one of the image is set, the lower/upper range in all the rest images are set to the same lower/upper range. # The rangeValues are always reset each time when the Color Stretch dialog is open. ",5 DM-6119,"update ""newinstall.sh"" nebula images & docker containers - w_2016_20",NULL,1 DM-6121,"Remove old DipoleMeasurementAlgorithm from imageDifference.py","Currently the new algorithm is run alongside the old. This ticket will deprecate the old algorithm, making the new one the default. This will be done after the new algorithm is vetted on real data (DM-5412). It may also be blocked by the pending SFM overhaul so that it can be implemented as a standard registered plugin.",4 DM-6123,"Build SFM housing for PSF approximation using ngmix code","Build a measurement plugin which allows PSF approximation to be done using ngmix. After consulting with Erin, it was decided that this would make use of the EM code and would produce as its output some variable number of Gaussians. These will be turned into MultiShapeletFunction outputs. A suitable set of configuration options and output failure flags will also be provided.",6 DM-6124,"Testing ngmix Psf plugin with CModel","Test that the ngmix PSF approx plugin works correctly in our measurement framework by testing it with CModel and comparing results with those produced with ShapeletPsfApprox.",6 DM-6125,"Do robustness tests of ngmix PSF approx plugin","Run tests on the ngmix PSF approx plugin similar to those which were run on ShapeletPsfApprox. We will test both for how long the plugin takes to run, and how often it fails. Note previous report on CModel and SPA was DM-4368",6 DM-6126,"LSST's version of Astrometry.net doesn't build on Ubuntu 16.04","Reproduced building on Ubuntu 16.04. https://groups.google.com/forum/#!topic/astrometry/aDCjhfMYhpE The current version (0.67) does build successfully standalone. These two patches fix 0.5.0: https://github.com/dstndstn/astrometry.net/commit/7ded70917d7cf1efa1d3af6d0da8b336ebbf9d92.diff and https://github.com/dstndstn/astrometry.net/commit/7c65b3cefc4f33c59af90c1a40b5f246002cdf28.diff Though only the first one is needed, I believe the second one is part of the build already.",1 DM-6127,"ngmix has no license","ngmix does not have a license, which means we shouldn't distribute it. Work with Erin Sheldon to see if he is willing to add one.",1 DM-6128,"Expanded view not doing fit/fill consistently ","Expanded view not doing fit/fill consistently. Sometimes is seems to fit/fill and resize it correctly, other times it stays at the zoom level. It should always fit/fill and change zoom level with resize when in expanded mode. (unless zoom type is FORCE_STANDARD).",2 DM-6130,"Fix docker git script: providing both -R and QSERV_DIR make it fails",NULL,2 DM-6133,"mpi4py does not compile under Yosemite due to hardcoded MACOSX_DEPLOYMENT_TARGET","{{mpi4py}} build on Yosemite (Mac OS X 10.10) fails with {code} _build.log:[2016-05-17T16:51:55.847161Z] error: $MACOSX_DEPLOYMENT_TARGET mismatch: now ""10.9"" but ""10.10"" during configure {code} For details see attached build log. The {{MACOSX_DEPLOYMENT_TARGET}} is being set in {{ups/eupspkg.cfg.sh}} {code} [serenity mpi4py] cat ups/eupspkg.cfg.sh # If MACOSX_DEPLOYMENT_TARGET is not set, we force it to be at least 10.9 # (Mavericks). This is the earliest version of OS X expected to work with # release 11 of the LSST stack. # This works around DM-5409, wherein mpi4py was attempting to use an OS X 10.5 # SDK, based on querying Anaconda, and failing. export MACOSX_DEPLOYMENT_TARGET=${MACOSX_DEPLOYMENT_TARGET:-10.9} {code} What is it that is supposed to be setting {{MACOSX_DEPLOYMENT_TARGET}}? And why is it not set at the time when {{ups/eupspkg.cfg.sh}} is run, but is set to 10.10 by the time the actually compilation is done? ",1 DM-6134,"Fix style of catalog panel "," finish up DM-5388 ticket by fixing the UI style of the panel",4 DM-6135,"Migrate VO search panel","VO search panel is a tab part of the catalog drop down panel that should be migrated. This was an outlier of DM-5388 ticket.",16 DM-6136,"Review and connect validation part to the input area field component","Catalog panel (DM-5388) needed an input area for polygon input search but it needs a review and connect the validation reducer to it.",6 DM-6137,"Add input based on catalog DD table","Add a panel to use DD catalog information to allow user to input catalog constraints as editable table component. This is to get a table with one or more extra column which are input field. The extra column needs a different default cell renderer (TextCell) - see TableRender.js. This table component needs to be hooked up to the fieldgroup or some way so it can be used in the catalog search panel. The catalog search panel will need to be adapted to display this table constraints to fully complete the search query options. The implementation: It contains the redesigned panel based on IRSA current OPS catalog search. A table with input constraints and sql area input are added. There is a lot in that PR. In particular, usage of table renderers and 'fieldgroup' together with a changed BasicTable component to be able to get the value from input field custom column 'constraints'. The panel will be reviewed also by IRSA and some input requirements were left out for now (mainly aesthetic details). Another left-out is the text are component and the validation of it. That should be addressed in the other ticket DM-6136. ",12 DM-6138,"Change Fields groups to handle other actions better","The fields group can be out of sync with actions if they are trying to use store data when that actual value is changes. This is a classic side-effect issue. It can be solved with sagas. Our current example. The color panels updating from the plot when the activePlotId changes. More to do: * field groups need a sega to more effective respond to out side actions * the dispatchChangeFieldGroup needs better, more documented parameters * update multiple fields at the same time. * should we have the field group support reset to init state? probably not, but look into it. * change init values? * Check example dialog and see if the large/smaller example is validating correctly.",2 DM-6139,"Change server side hardcopy code to work better with the non-GWT call","The server hard to make a hard copy now takes a StaticDrawInfo object. We want to use only a region array. Change the server side to support this.",2 DM-6140,"Produce tech note describing detailed project management procedures","Write a technical note describing the detailed project management procedures derived by the pmp-wg. Source material is [~jbecla]'s document at https://github.com/lsst/ldm-pmt/.",8 DM-6141,"Drawing layer improvement to handle mouse selection","Drawing layers are not handling and sharing the mouse quite right. Also the mechanism to determine to is priority for the mouse needs work as well. This is all necessary to make markers work correctly, since every marker is an individual drawing layer. Also, the draw layer utilities are all in PlotViewUtil.js. They need to be moved to something closer to the draw layers.",4 DM-6142,"Client side Hardcopy support for png with drawing layer overlay","Add all hard copy support so we can create a png with all the overlays.",6 DM-6145,"jenkins/qa terraform destroy fails if there is an existing rds final snapshot","AWS appears to prevent the overwrite of an existing final rds snapshot. This scenario may arise when creating/destroying a dev env multiple times. This can be avoided by disabling the final snapshot when destroying an rds instance. One way to resolve this would be to add a terraform var to signal this is his is a development env.",2 DM-6146,"Evaluate performance of dipole fitting in crowded regions","There are legitimate concerns about performance of the new dipole fitting algorithm in crowded fields. This will be evaluated (on real data? if no existing data, then realistic simulated data) and contrasted with other possible alternatives. This is in response to Zejlko's concern and suggestion that DipoleFitTask should constrain only positions using pre-subtraction images, and only fit fluxes using the diffim.",8 DM-6147,"Set SUSPECT mask in ISR task and make saturation a double","Implement RFC-190 and mask suspect pixels: Add {{selectLevel}} to {{lsst.afw.cameraGeom.AmpInfoCatalog}}, as a double, and add support for it to {{lsst.ip.isr.IsrTask}}, analogously to masking saturation: iif {{suspectLevel}} is not {{nan}} then set the {{SUSPECT}} flag for pixels above the suspect level. Also change the type of {{saturation}} in the {{AmpInfoCatalog}} from {{int}} to {{double}}, so that the existing test for {{nan}} actually works",5 DM-6149,"Reduce memory utilization in mysql proxy","Jon is trying to run tests with large result which kills proxy/czar because it runs out of virtual memory. Would be nice to reduce memory use and find a way not to keep query result in memory.",2 DM-6151,"Failure to fail when fallbackFilterName is None","When no {{fallbackFilterName}} is set, we can get a confusing error message when failing to load a calib: {code} RuntimeError: Unable to retrieve dark for {'filter': 'U', 'date': '2016-05-12T02:58:56.591', 'ccd': 0, 'basename': '2016-05-12skyflats_02', 'object': 'FLAT', 'visit': 883, 'expTime': 20.0006, 'channel': 16} and no fallback filter specified: Unknown value type for filter: <type 'NoneType'> {code} This is unrelated to the calib load failure, and merely reflects the fact that {{fallbackFilterName=None}}.",1 DM-6153,"long term re-plan","We need to make the long term plan for FY2017 - FY2019, ready for ComCam; and more FY2020- FY2022, ready for operation.",30 DM-6154,"New features in histogram ",". 1D histogram, maybe a special case of density plot . new way to calculate the bin size? ",10 DM-6155,"Attend SciPi WG meeting","Attend the Science Pipelines Working Group meeting in Seattle.",8 DM-6156,"Attend SciPi WG phonecon","Attend the Science Pipelines working group meeting by video con.",3 DM-6157,"Flesh out MOPS work","Work with Lynne Jones and Colin Slater to flesh out the high level MOPS design to a point where we can plane the risk associated with each component.",3 DM-6158,"Attend SciPi WG F2F in Tucson","Attend the Science Pipelines working group face to face meeting in Tucson.",3 DM-6159,"Flesh out the Level 1 processing diagram","Andy and I need to make sure we understand the Level 1 processing.",10 DM-6160,"Adding an int to the end of CzarConfig causes a segfault error.","Adding and int to the private members of CzarConfig causes a segfault when Czar::Czar() calls LOG_CONFIG(logConfig);. gdb shows logConfig is the correct string value but somewhere in log4cxx something is corrupted and causes a segfault. Adding an int to the end of Czar (the class where CzarConfig is instantiated) does not cause the issue. ",6 DM-6162,"Investigate how the diffim decorrelation correction works for the case of non-uniform PSFs and noise","It is not clear whether, or how, the L(ZOGY) post-convolution kernel (PCK; see DM-5914) will work for non-uniform PSFs or noise/variance. This will be investigated using the simple implementation from DM-5914. Tasks: 1. Determine whether variation in the PCK across the field is significant enough to matter for typical LSST images 2. If it does matter, investigate options for performing interpolation of the PCK across the field, or via calculating the PCK across the field from the spatially-varying matching kernel. ",8 DM-6165,"Extend the capabilities of the StarFast simulator beyond the minimum needed for DCR algorithm testing","During development of the StarFast simulator for DCR correction algorithm testing several addition features were identified that would make it more general and useful. These capabilities will enable a wider range of testing of new image differencing algorithms and give greater confidence in the accuracy of the results for DCR simulations. ",14 DM-6166,"Time AST and compare to our WCS code","Time TAN-SIP for our code and for AST, in order to get a sense of the performance impact of switching to AST for our WCS implementation.",3 DM-6167,"Create DMBP project in jira","Create a new project in JIRA (DM Baseline Plan), spec provided here: https://confluence.lsstcorp.org/display/DM/ProjMgmtWG%3A+The+New+DLP I am sure we will fine tune it, but it is a (hopefully good) start.",2 DM-6168,"Wrap afw using pybind11","Experiment with using pybind11 (rather than Swig) to expose afw, and the packages it depends on, to Python. The concrete result of this epic is an assessment of the utility gained by wrapping the rest of the stack in pybind11 and an estimate of the time that would be required to carry out that work. If those goals are reached without completing the work on afw, we can claim success. In particular, if it becomes clear early in the epic that there is no long term utility here, we should abort the rest of the work.",80 DM-6169,"Participate in DM replanning process","Participate in the ongoing DM replanning process. This includes contributions to both the scipi-wg and pmp-wg, including such documentation writing or other tasks as the chairs of the those groups request, as well as resource loading and delivering the complete plan.",100 DM-6170,"PSF fitting study","Investigate the [Piff|https://github.com/rmjarvis/Piff/] PSF modelling code. Experiment with applying it to realistic LSST (or precursor) data. Discuss whether it is an improvement over existing techniques, and a recommendation as to whether it should be adopted in the stack. (NB writing the code to incorporate it into the stack is not a requirement of this epic, but may be a desirable side-effect). If Piff is still too immature for this to be useful, investigate Kendrick Smith's [hscpsf|https://github.com/lsst-dm/meas_extensions_hscpsf/] code.",15 DM-6171,"Service technical debt accumulated in earlier cycles","Service technical debt accumulated in earlier cycles.",100 DM-6172,"F16 DRP emergent work","Handle emergent work during F16.",55 DM-6173,"Serve as chair of the IVOA Time Domain Interest Group","[~swinbank] will serve as chair of the IVOA Time Domain Interest Group through May 2017. This epic captures work associated with that activity in F16.",20 DM-6174,"Prepare for and participate in SBAG meeting","By request of [~zivezic], [~nlust] will participate in the June 2016 meeting of the [Small Bodies Assessment Group|http://www.lpi.usra.edu/sbag/meetings/]. This epic captures work associated with preparation for that meeting.",20 DM-6175,"Visualization tools for Science Pipelines","[~nlust] will collaborate with the SUIT group on development of appropriate visualization tools in support of the work of the Science Pipelines group during F16. [~xiuqin] will provide further specification of success criteria.",20 DM-6176,"Optimal coaddition","Experiment with building ""optimal"" coadds, as defined by [DMTN-015|http://dmtn-015.lsst.io/en/latest/]. The aim here is to be able to generate coadds for experimentation with measurement algorithms. The expected output is appropriate mathematical formalism and prototype code. Polished integration of this facility with the LSST stack is not a requirement (but may be a useful by-product) of this work.",100 DM-6177,"Increase memory locked amount in container","In order to lock memory, the memory locking limit within the container for the qserv worker needs to be raised. My understanding is the container uses whatever is the host setting so the limit has to be set for the container user and whatever the user is inside the container. The particular limits is: memorylocked 64 kbytes notice that by default it's 64K. That needs to be raised to say 75% of the real machine size. I wouldn't make it unlimited as a memlock mistake may crash the whole machine. The limits are specified in ""/etc/security/limits.conf"". You will know that you are successul when you ssh into the container as the qserv worker user and the ""limit"" command tell you have can lock lots of memory. We would also set the CAP_IPC_LOCK privilege but setting the soft/hard limit above should be good enough. So, let's start with that. ",2 DM-6178,"Add eups version for Qserv for stack package version","For stack packaged Qserv version, version needs to be retrieved and added to monitor.yaml using next command: dev@clrinfopc04:~/src/qserv$ eups list qserv -s -V LOCAL:/home/dev/src/qserv Indeed, pkgautorversion doesn't work in this case, I.e. with no git repos",2 DM-6179,"Support Python 3 migration","Support the migration of the DM code to Python 3. This includes writing transition documentation, integration of a new scons, migrating a handful of low-level packages and liaising with the teams on their packages. The final outcome of this epic is that everything would be in place for the migration at the August All Hands meeting.",40 DM-6180,"Update LSE-61 requirements and traceability","With the updates to DPDD and LDM-151 in the early part of F16, there is a need to update LSE-61 (DMSR) such that it can directly trace requirements from OSS+DPDD through LSE-61 and down to implementation LDM documents. This will require substantial rewrites of many of the existing requirements and possible addition of new requirements. It may also be necessary to add annotations to DPDD and other LDM documents to provide traceability anchors for DMSR. The outcome of this epic is a new baselined DMSR approved by CCB.",40 DM-6181,"reST roles for JIRA References","Add {{:jira:`DM-1234`}}-type roles to documenteer so that JIRA tickets, epics and RFCs can be referenced easily from all of our Sphinx-based projects.",2 DM-6183,"sourceSelector needs a schema in ImageDifferenceTask","imageDifference.py crashes with a vague error on initializing the sourceSelector task. The problem turns out to be that sourceSelector needs a schema passed in.",1 DM-6184,"Add a python 3 Jenkins instance","We need a Jenkins instance where the default python in the PATH is python3 (where version >= 3.4 with 3.5 preferred). The underlying OS does not matter. A prerequisite of this is a modification to the {{lsstsw}} {{bin/deploy}} script to allow Python3 to be installed ({{miniconda3}} EUPS package?) rather than python2. Modifying the EUPS {{scons}} and {{python}} packages is outside the scope of this ticket. A build of a third-party EUPS package is sufficient demonstration of the capability.",12 DM-6185,"Get jointcal running on minimum data","It is very important for other teams to have a version of jointcal running to remove the sensitivity on the errors in astrometric reference catalogs. The suggestion is to get jointcal running with CFHT, HSC, DECam and lsstSim.",100 DM-6186,"Provide input for the update of LDM-151","We need to update the Level 1 portions of LDM-151 to be both more descriptive and close to what we actually plan to deliver. This will involve breaking down things to a finer level of planning as well as delivering content for the document.",38 DM-6188,"First draft of overview (""vision"") document","See https://dmtn-016.lsst.io",3 DM-6189,"Complete update to LDM-230","Complete an update to LDM-230 and submit to TCT for re-baselining. Along the way, contribute to and review other documents needed by DPS-WG.",10 DM-6190,"Update sizing/cost model","Contribute to the updating of the sizing/cost model, including fixing known bugs, synchronizing the inputs and models to fit the current baseline, and investigating changing the modeling technology.",10 DM-6191,"Refine SuperTask design document","Deliver a refined SuperTask design document including reslicing to accommodate changes in the axis of parallelization between SuperTasks making up a composite SuperTask.",13 DM-6192,"Update LSE-75 DM-TCS ICD","Submit an LCR to update LSE-75 to reflect current thinking on telemetry feedback from DM to the TCS.",5 DM-6193,"Update LSE-72 DM-OCS ICD","Submit an LCR to update LSE-72 to reflect changes discovered by work at NCSA to support early integration tests.",5 DM-6194,"Update LSE-68 DM-Camera DAQ ICD","Submit an LCR to update LSE-68 to reflect understandings developed between Mike Huffer and NCSA about the interface, including the image deletion policy for the camera data buffer.",5 DM-6195,"Provide input to Commissioning Plan","Provide input based on understanding of the DM interfaces to the Commissioning Plan being developed by Chuck Claver.",5 DM-6196,"SQuaSH capability extension: multiple testdata service","This epic covers work to deliver the following improvements to the SQuaSH prototype stood up in X16: - drilldown 1 level (time series->histogram) - multiple testdata options (requires jenkins, backend, dashboard extension) - processccd + validate_drp pseudo-workflow - pseudo-provenance (track manifest.txt - real LSST provenance system will be swapped in for extensive functionality when available) ",92 DM-6197,"Update LSE-76 Summit ICD","Submit an LCR to update LSE-76 based on Summit rack and power needs obtained from Ron Lambert.",2 DM-6198,"Ad-hoc Docs & Comms requests","Timeboxed epic for in-cycle ad-hoc developer or management requests. In the first half of FY16 most of these are likely to be deveoper-guide related. ",12 DM-6199,"Stack API documentation ","Stack API Doc generation -> pipelines.lsst.io",20 DM-6201,"Resource load F16 part II","Resource load for second half of F16 (SP estimate from first half)",6 DM-6202,"SQuare Requirement, Design, & Review Docs for DM","This is an epic to track the work required on for DM baselined documentation from SQuaRE staff, including any associated with Working Group / replan etc [FE: 45% MWV: 45% DN: 10%] ",35 DM-6203,"Releases and Release Engineering Improvements"," [50% FE 50% MJP]",16 DM-6204,"Build/CI/Deploy improvements requested by the DAX/Qserv team"," Requests for Build/CI/Deploy improvements initiated by the DAX/Qserv team prioritised on request from the DM Project Manager. ",24 DM-6205,"Build/CI/Deploy improvements requested by the Architecture Team"," Build/CI/Deploy improvements requested by the Architecture Team prioritised by request from the DM System Architect. They cover predominantly support for the Python3 support. ",4 DM-6206,"CI Improvements: Jenkins 2 upgrade etc"," This epic covers a timeboxed maintainance of the Jenkins-based CI system, including the Jenkins 2 upgrade as well as the required updates to the Jenkins-puppet module. It also may include work done as part of DM-6204 brought over to the apps CI service. ",8 DM-6207,"CI/Build/Deploy improvements for Sims","This is a timeboxed effort to prioritise support requests from the Sims group",4 DM-6208,"SQuaRE services disaster recovery","This is a timeboxed effort to test and improve backups and disaster recovery for SQuaRE services. It is unlikely to be sufficient in itself. ",8 DM-6209,"Ad-hoc developer requests","This is a bucket epic for ad-hoc developer requests that cannot be postponed till the next planning cycle. In the event that it is underutilised for this purpose, it will be assigned to technical debt DM-5850",8 DM-6210,"Improve OSX support",NULL,16 DM-6211,"Gitlfs maintenance - protocol upgrade etc",NULL,8 DM-6212,"Slack migration",NULL,8 DM-6213,"Conda binary distribution improvements",NULL,16 DM-6214,"logging.lsst.codes improvements",NULL,8 DM-6215,"Verification dataset exploratory work","[DN 50% AF 50%]",16 DM-6216,"F16 DAX Services Containers & Ops",NULL,10 DM-6217,"F16 DAX Services Improvements",NULL,10 DM-6218,"F16 NCSA Dax Services Deploy",NULL,10 DM-6219,"F16 Replan",NULL,33 DM-6221,"F16 Support SUIT for Prototype DAC",NULL,40 DM-6222,"F16 L1 DB Prototype I",NULL,89 DM-6223,"F16 NCSA Stripe 82 Image Ingest",NULL,36 DM-6224,"F16 Butler Repository Refactor","Per KT, the parent/peer repository relationship scheme was not an exact fit for what we need. We discussed and decided that butler should manage its own input and output repositories. Also discussed with KT and Gregory was the ability to select inputs by 'tagging' repositories. The design discussion with the larger group is captured in RFC-184.",40 DM-6225,"F16 Butler Storage & Format Refactor","We want a pluggable architecture that allows code that uses butler to be able to define the the storage format and location from configuration and/or run time code. (maybe it's implicit in this epic, but we need to define, design, RFC, and implement this feature.)",35 DM-6226,"F16 Butler Composite Dataset Design","Do design, RFC, and some prototype code for loading and saving ""composite datasets"" via butler. Composite Dataset definition: a python objects loaded by butler from file/database/etc that is persisted in more than one physical location (e.g. more than one file on disk). Those objects should also be able to be written to more than one physical location - the design should support this but the initial implementation may not be required to have this. ",50 DM-6227,"F16 Butler Repo of Repos Design",NULL,20 DM-6228,"F16 VO Standards Investigation",NULL,10 DM-6230,"F16 Qserv Loader Improvements",NULL,100 DM-6231,"F16 Qserv Containers and Ops",NULL,16 DM-6232,"F16 QServ Improvements",NULL,82 DM-6233,"F16 NCSA Qserv Deploy",NULL,34 DM-6234,"F16 NCSA Stripe 82 Catalog Ingest",NULL,20 DM-6235,"Take part in LDM-151 Progress Meeting, 2016-05-27",NULL,1 DM-6236,"Take part in LDM-151 Progress Meeting, 2016-05-27",NULL,1 DM-6237,"Take part in LDM-151 Progress Meeting, 2016-05-27",NULL,1 DM-6238,"Familiarization with RHL calibration documentation",NULL,4 DM-6239,"The grid labels are not placed in the right position when the coordinate is Ecliptic coordianates","The algorithm to calculate the label position does not work well for the Ecliptic coordinate system. The algorithm needs to be modified to work for all the coordinates.",2 DM-6240,"Support API interaction with Regions","We now need more fine grain controls over regions: From API, user can: * load region file * delete region layer * add a region entry to a layer * delete a region entry from a layer When this ticket is complete, region conversion should be completed.",10 DM-6241,"Implement the ZOGY extension to the A&L algorithm in the stack","DM-5422 provided a test implementation of the real space extension to the A&L algorithm for a correction kernel motivated by the ZOGY paper. This epic is to take that test algorithm and incorporate it so that it can be used by the diffim tasks. The first step will be to incorporate it using a static PSF t compute the correction kernel. The second will be to evaluate how that affects the resultant difference image. This should also include an estimate of the overall performance relative to the base A&L algorithm by examining runtime, false positive rate, and accuracy of noise estimation in both the detection threshold and the reported measurement SNR.",30 DM-6242,"Study spatial variability of ZOGY correction","DM-6241 looked at how the correction term to the A&L algorithm performs under the simplifying assumption that the science image PSF is spatially invariant (though the matching kernel is spatially varying). This epic will focus on how to extend the correction to include spatially varying terms.",38 DM-6243,"Study the impact of having a spatially invariant decorrelation correction factor to A&L","The initial implementation of the A&L + noise whitening correction term assumes a single matching kernel and variance value(s) for the image(s) in the correction kernel. We should assess how well that assumption performs in simulated and real images. One test would be the variance and covariance in the noise as a function of position in a set of typical images.",8 DM-6244,"Assess performance of the decorrelation correction to A&L","Study the performance when using the (currently, spatially invariant) correction term to the base A&L algorithm in terms of runtime, detection threshold, reported measurement noise, and false positive rate for similarly tuned versions of both the base algorithm and that with the correction applied.",6 DM-6245,"Compare competing algorithms for correcting DCR in template images","DM-5455 provides an implementation of a correction algorithm that depends on a matrix inversion approach to correct for DCR. This should provide another approach for comparison (potentially a more forward modeling based approach). Compare the algorithms in a simplified system in 2-D where DCR is along one axis. The algorithms should be extended to arbitrary rotations. The bakeoff will be repeated in the case of arbitrary rotations. The result should be a recommendation as to the algorithm to use for DCR correction in template images.",49 DM-6246,"Vertical overscan off by one again","In DM-5524 [~price] fixed the vertical overscan by directly editing the amp info catalogs, but didn't mark the camera generating code as bad. In DM-6147 I regenerated the files, reintroducing the problem. The problem seems to be a subtle bug in the camera generating code. Rather than try to fix it, I'll convert the fixed catalogs directly and mark the generating code as broken. [~price] will issue an RFC that suggests a better way to handle generating amp info and once that is dealt with we can come up with a more permanent fix (e.g. delete the generating code or fix it).",1 DM-6247,"DRP Outline for LDM-151","Write outline for Data Release Production section of LDM-151, using the DRP Data Flow diagram as the organizing principle.",2 DM-6248,"DRP Top-Level Diagram and Descriptions, Draft 1","Insert the content from the DRP Data Flow diagram on Confluence into LDM-151, adjusting it to the outline developed on DM-6247.",2 DM-6249,"Implement competing algorithm","Implement a competing (potentially a forward modeling approach) algorithm for correcting DCR in templates.",12 DM-6250,"Extend competing algorithm to arbitrary rotation angles.",NULL,6 DM-6251,"Convert DRP Top-Level Diagram to standard conventions","DM-6248 adds a large, complex diagram that will need to be cleaned up and converted to use the same conventions and colors as other diagrams in LDM-151.",2 DM-6252,"Do bakeoff between the two algorithms in simplified case","The original matrix inversion technique and the competing technique will likely have different sensitivities. This should be a comparison of the algorithms, likely based on numbers of dipoles, along with performance (memory and runtime) considerations. This will be done on 2-D images with DCR along one axis.",6 DM-6253,"Bakeoff between algorithms extended to arbitrary rotation.","Redo bakeoff in the case of arbitrary rotation in DCR effect.",6 DM-6254,"Develop standard conventions and colors for LDM-151 diagrams","We want diagrams in LDM-151 to have consistent notation and colors, and to be produced using the same tool. Someone needs to look at the diagrams produced so far to gather requirements, decide on and document these conventions, and select the tool we'll use to produce them.",4 DM-6255,"Improve detail for for DRP imchar/jointcal in LDM-151","Write more detailed descriptions and possibly draw a rough diagram for the single-frame processing and simultaneous calibration components of Data Release Production. Does not necessarily involve turning this section into prose.",6 DM-6256,"Improve detail for DRP background matching, coaddition, and diffim in LDM-151",NULL,4 DM-6257,"Improve detail for DRP coadd processing in LDM-151",NULL,2 DM-6258,"Improve detail for DRP object characterization in LDM-151","Includes coadd measurement, multifit, and forced photometry. Could be faster to write than other sections because we can lift from ""blended-measurement"" document that already exists in LDM-151 repo; could be harder because that document has already exposed a number of unresolved questions that may need to be addressed (by at least getting agreement among pundits on the best-bet approaches) before we can plan.",4 DM-6259,"Improve detail for DRP afterburners and level-3 gathering in LDM-151",NULL,2 DM-6260,"Cleanup and standardize DRP imchar/jointcal diagrams","DM-6255 will produce some rough, draft-level diagrams that will need cleanup and standardization.",1 DM-6261,"Cleanup and standardize DRP background matching, coaddition, and diffim diagrams","DM-6256 will produce rough diagrams that will require cleanup and standardization. [~ctslater] has made some suggestions for the current diagram that I'll implement on this issue, so I'm assigning it back to me. I'll also go ahead and integrate his updated DRP overview diagram (currently on Confluence) into LDM-151 here. ",1 DM-6262,"Cleanup and standardize DRP detection, association, and deblending diagrams","DM-6257 will produce rough, draft-level diagrams that will require cleanup and standardization.",1 DM-6263,"Cleanup and standardize DRP object characterization diagrams","DM-6258 will produce rough, draft-level diagrams that will require cleanup and standardization.",1 DM-6264,"Cleanup and standardize DRP afterburners and level-3 gathering diagrams","DM-6259 will produce rough, draft-level diagrams that will require cleanup and standardization.",1 DM-6265,"Audit DRP LDM-151 for correct handling of chromaticity","Correctly handling wavelength-dependent photometric and PSF effects is one of the biggest qualitative differences between the current state-of-the-art and what we have in mind for LSST, and that makes it easy to get wrong. We need to make sure all steps that produce high-quality fluxes or rely on high-quality PSFs have access to object colors and a reasonable approach to using them. ",2 DM-6266,"Upgrade cfitsio and deal with long keyword handling","To implement RFC-105, we need to figure out how we are handling long FITS header keywords, before we can upgrade to cfitsio 3.38 or newer. There may be other FITS-related idiosyncrasies in the stack that may be brought to light while upgrading, as 3.38 has changed how it handles some of the non-standard conventions. See some of the notes in DM-4115 for problems encountered while attempting to upgrade to the 3.38 beta.",20 DM-6269,"Attend HTCondor Week","Attend HTCondor Week with [~gdaues] to learn about condor and pegasus http://research.cs.wisc.edu/htcondor/HTCondorWeek2016/index.html",20 DM-6270,"Review of Workflow Systems","Review different workflows and write a final comparison report. The plan is to look at up to 8 workflows. Current list of workflows: - pegasus, HTCondor - panda - swift - ... Each review should take around 3 days. The goal is to review the workflows systems with: - longevity, how long has the system been around, what is the funding? - use cases, who is using it? - scale, how large of a workflow has it used? - code, is the code open, how is the developer community, does it have python bindings? - GUI, does it have some easy way to monitor the workflow? - can we generate workflows programmatically? - what clusters are supported? Ideally we should also try and get it up and running and maybe even generate a dummy workflow. Discussion could be how we can prototype DRP and use that as a use case. Deliverable: Evaluation report Staff: Rob Kooper, Hsin-Fang Chiang, Matias Carrasco Kind, Mikolaj Kowalik, Steve Pietrowicz Effort: 25 days Planned Start: 6/1/2016 Planned End: 6/30/2016",50 DM-6271,"Audit DRP LDM-151 for correct handling of crowded fields","[~jbosch]'s background is in extragalactic science on high-latitude fields, and he frequently forgets to think about how algorithms will perform in crowded stellar fields. When the first draft is complete, we should have someone experienced in that area read closely to check that he hasn't made any incorrect algorithmic assumptions as a result.",1 DM-6272,"Reception and Placement",NULL,7 DM-6273,"Statement of Work ",NULL,12 DM-6274,"Propose extension of SuperTask functionality for workflow package","Should see what is needed to add to supertask so we can use it with workflows. After this is decided we should create a RFC for implementation. Deliverable: RFC to extend SuperTask Staff: Rob Kooper, Matias Carrasco Kind, Mikolaj Kowalik Effort: 5 days Planned Start: 6/1/2016 Planned End: 6/22/2016",10 DM-6275,"Implementation of Supertask RFC","This should implement the RFC written in DM-6274. Note that this activity is independent of the work to complete the supertask and activator prototype in DM-6418. Deliverable: Deliverables based on outcome of RFC Staff: Matias Carrasco Kind, Mikolaj Kowalik Effort: 15 days Planned Start: 7/1/2016 Planned End: 7/31/2016",30 DM-6276,"ConOps for Workflow/Middleware","Create a conops for workflow, this will depend on some decisions made about L2 conops. Deliverable: ConOps document for Workflow Staff: Rob Kooper, Hsin-Fang Chiang, Matias Carrasco Kind, Steve Pietrowicz, Jason Alt, Margaret Johnson Effort: 15 days Planned Start: 7/1/2016 Planned End: 7/31/2016",30 DM-6277,"Proof of Concept Implementation of Workflow System","This should start the implementation of the workflow system. This will be a proof of concept only. Deliverable: Proof of concept workflow implementation code Staff: Hsin-Fang Chiang, Mikolaj Kowalik, Rob Kooper, Steve Pietrowicz Effort: 20 days Planned Start: 8/1/2016 Planned End: 8/31/2016",40 DM-6278,"Investigate proper precision for afw::image::Image pixel transforms","The various pixel based transforms in {{afw/src/image/Image.cc}} were converted from using {{boost::lambda}} to C++11 lambda per DM-6091. At many places the previous implementation contained implicit casts (through {{boost::ret}}) of intermediate results to {{PixelT}} (e.g. {{float}}). In particular this affects opperations such as {{result = l + c*r}} where {{l}} is the left hand side image, {{r}} is the right hand side image and {{c}} a {{double}} constant. When calculated at double precision (e.g. without the casts, which are not needed with C++11 lambdas) the result is slightly different and this causes {{tests/testProcessCcd.py}} to fail on {{self.assertAlmostEqual(psfIyy, 2.17386182921239, places=7)}} which is only equal up to the fifth place. In order to not break existing behaviour I added explicit casts to {{PixelT}} for intermediate results. But this approach is questionable as the end result will be less accurate then possible. The aim of this ticket is to decide which approach is best: 1. Calculate at full precision and modify the test case. 2. Cast intermediate results to final precision (as it is done now). 3. Do something else?",1 DM-6279,"Fix possible logic error in pex_policy dictionary","Investigate and fix the following warning in {{pex_policy}}. {code} src/Dictionary.cc:312:9: warning: logical not is only applied to the left hand side of this comparison [-Wlogical-not-parentheses] if (!getType() == Policy::POLICY) // should have checked this at a higher level ^ ~~ src/Dictionary.cc:312:9: note: add parentheses after the '!' to evaluate the comparison first if (!getType() == Policy::POLICY) // should have checked this at a higher level ^ ( ) src/Dictionary.cc:312:9: note: add parentheses around left hand side expression to silence this warning if (!getType() == Policy::POLICY) // should have checked this at a higher level ^ ( ) src/Dictionary.cc:312:20: warning: comparison of constant 'POLICY' (5) with expression of type 'bool' is always false [-Wtautological-constant-out-of-range-compare] if (!getType() == Policy::POLICY) // should have checked this at a higher level {code}",1 DM-6280,"The labels in HMS formate are wrong in WebGrid","The labels in HMS format no longer show hh:mm:ss anymore. The porting introduced the bug. ",1 DM-6283,"Fix mismatched-tags warnings in meas_modelfit","The following warnings are produced in {{meas_modelfit}}. Fix them. {code} include/lsst/meas/modelfit/UnitSystem.h:90:1: warning: 'LocalUnitTransform' defined as a struct here but previously declared as a class [-Wmismatched-tags] struct LocalUnitTransform { ^ include/lsst/meas/modelfit/Model.h:41:1: note: did you mean struct here? class LocalUnitTransform; ^~~~~ struct {code}",1 DM-6284,"Remove swig special casing for obsolete boost features","In {{utils}} the file {{python/lsst/p_lsstSwig.i}} defines special cases for boost features that are removed as part of DM-5880. This ticket removes the special cases.",1 DM-6285,"Chart API: external API and the API for histogram","need to support external API and the API for histogram for now only addXYPlot and showPlot are implemented",4 DM-6286,"Charts (XY plot, histogram) Container","Need to be able to view multiple charts simultaneously in the chart area User would like to see multiple XYplots from the same data displayed at the same time. One example will be to display different color-color plots using all 4 bands data from WISE catalog. It is also possible we want to display histogram data at the same time as color-color plots.",10 DM-6287,"Charts refactoring","I'd like to do some cleanup, which would facilitate further development. This includes: - moving chart related code to a separate package (now it is in visualize) - converting components created with React.createClass to es6 classes - reorganize store and controllers to have all charts related things under 'charts'. Now we have 'charts' for charts ui, xyplot for xyplot charts, histogram for histogram charts, and tblstats for table statistics. Fixed bugs * missing chart mount action, when a chart is removed and then recreated on the same table Steps to reproduce: load a table (default scatter plot created), create histogram, delete scatter, create new scatter. The last scatter did not produce mount action, and the plot was not tracking table changes, like filter. * undefined shows as a label when no server call is necessary Steps to reproduce: load table (default scatter created), clear options and choose the same columns , click apply. ""undefined"" are shown as axis labels",4 DM-6288,"Chart options display","Make chart options ""in-place"" popup, similar to table options for consistent look. It will also alleviate resizing, because the chart size won't need to change when options are open.",3 DM-6289,"Chart options reset and clear","Need to support reset and clear for plot and histogram options. Should be no-brainer after DM-6138 (update multiple fields at the same time)",3 DM-6290,"Attend SBAG Meeting","Meeting runs Tues 28 to Thurs 30 June; that means we'll likely lose Nate for the whole week, given travel.",8 DM-6291,"Read materials related to SBAG prep and attend telecon","Read up material to prepare for SBAG, and discuss readings with Mario, Lynne, and Zeljko.",4 DM-6293,"Fix error in cmodel related to computing LinearTransforms","When running cmodel in ci_hsc, the cmodel plugin throws the error: {code} processCcd.charImage.detectAndMeasure.measurement WARNING: Error in modelfit_CModel.measure on record 775961510756221246: File ""src/geom/LinearTransform.cc"", line 66, in const lsst::afw::geom::LinearTransform lsst::afw::geom::LinearTransform::invert() const Could not compute LinearTransform inverse {0} lsst::afw::geom::SingularTransformException: 'Could not compute LinearTransform inverse' {code} This seems to be causing some aperture corrections to fail, as there are no sources to compute the corrections from. Investigate why this error is being thrown. If it is a bug, fix it, if the code is not handling situations it should then make the algorithm more robust.",4 DM-6294,"Add support for pybind11 to build system","Add pybind11 as third party package to the stack. Update sconsUtils to support building with pybind11. Use daf_base DateTime to demonstrate that this works.",8 DM-6295,"Unit test for coadds in pipe tasks detects too many sources","The unit test for pipe tasks creates a dozen stars, to use in coaddition testing. However the results of running the test show over a hundred sources found. Investigate why the extra sources are being detected, and fix to increase the robustness of the test. If this relates to other sections of the codebase (deblender) investigate if it is appropriate to make changes to those components to make them more robust instead of creating a simple hack in the unit test.",5 DM-6296,"Wrap afw::geom with pybind11","The generated wrappers will live parallel to the Swig wrappers. This ticket only covers the C++ wrappers themselves, not the Python layer on top (which will continue to use the old wrappers) all work will stay on a separate branch and will not be merged to master until DM-6168 is complete.",5 DM-6297,"Wrap afw::detection with pybind11","The generated wrappers will live parallel to the Swig wrappers. This ticket only covers the C++ wrappers themselves, not the Python layer on top (which will continue to use the old wrappers) all work will stay on a separate branch and will not be merged to master until DM-6168 is complete.",5 DM-6298,"Wrap afw::math with pybind11","The generated wrappers will live parallel to the Swig wrappers. This ticket only covers the C++ wrappers themselves, not the Python layer on top (which will continue to use the old wrappers) all work will stay on a separate branch and will not be merged to master until DM-6168 is complete.",5 DM-6300,"Extend galaxy shear fitting results to cover ngmix",NULL,10 DM-6301,"Write example meas_base plugin in Python","During X16, new functionality was exposed to Python plugins in meas_base. Write a complete pedagogical example. It should go beyond our current pure Python plugins to demonstrate use of: * FlagHandler; * SafeCentroidExtractor; * Other relevant, undocumented functionality. This should be added to the package level documentation for meas_base, so it appears in some extended version of https://lsst-web.ncsa.illinois.edu/doxygen/x_masterDoxyDoc/meas_base.html.",6 DM-6302,"Wrap pex_exceptions with pybind11","While wrapping these packages, pay particular attention to exception translation (see Jim's bullet point 3: https://jira.lsstcorp.org/browse/RFC-182?focusedCommentId=48644&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-48644).",4 DM-6303,"Wrap ndarray with pybind11","Note particularly Jim's second bullet point at https://jira.lsstcorp.org/browse/RFC-182?focusedCommentId=48644&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-48644.",12 DM-6304,"Wrap afw dependencies in pybind11","Everything that isn't base, utils, pex_exceptions (DM-6302), ndarray (DM-6303).",10 DM-6306,"Executable test in utils needs to test an executable","In DM-4036 all the test binaries were removed as no longer being needed. This had the unfortunate side effect that the {{testExecutables.py}} test no longer tests anything. This ticket will be used for adding a test file.",1 DM-6307,"input outlines to LDM-151 for AP","Write outlines into the LDM-151 document for the alert production pipelines.",4 DM-6308,"Upload JSON from validate_drp to SQuaSH REST API on Jenkins","This ticket covers work to build a Python package/script whose role is to take JSON output from validate_drp (DM-6086), shim it into the JSON schema currently expected by the SQuaSH REST API (http://sqr-009.lsst.io), and post the data to the API's {{/jobs}} endpoint. This tool also adds additional metadata to the ‘Job’ document, including the build ID and versions of packages as run by validate_drp (see DM-5943).",2 DM-6309,"Update LDM-151 with SDQA Skeleton and Outline","1. Update the LDM-151 draft with the SDQA Skeleton from the DMLT + SciPipelines working group discussions of May 16-20. Implement as bullet points in a semi-coherent list. (/) 2. Clean up list. (/)",1 DM-6310,"Transform SDQA bullets to prose",NULL,1 DM-6311,"Create 1st-level block diagrams for SDQA",NULL,2 DM-6312,"Update Scons to v3.0","Scons v3 is the scons version that supports both Python 3 and Python 2.7. This ticket is for updating Scons and ensuring that the Python 2.7 stack still builds. This work depends on the Scons developers delivering a new Scons by mid July. Whilst work is ongoing it may be necessary to help out with the port if we wish to meet our Python 3 target.",4 DM-6313,"Create miniconda3 EUPS package","{{newinstall.sh}} currently installs miniconda via EUPS. To replicate that functionality in Python3 we need to create a {{miniconda3}} package. This package should be almost identical to {{miniconda2}}. Requires that {{lsstsw}} first be updated to support python 3.",1 DM-6314,"Port lsstsw to Python 3","Get {{lsstsw}} working with Python 3: * Update the {{deploy}} script to allow a Python 3 python to be installed and modify the version checking code. * Demonstrate that {{lsstsw}} {{rebuild}} will successfully build and install a third-party non-Scons package.",5 DM-6315,"Write Python 3 porting guide","Porting the stack to Python 3 is not as simple as blindly running {{futurize}}. A guide has to be written explaining the issues and providing guidance on when to accept {{futurize}} suggestions and when to ignore them. This guide will be written as a Tech Note.",10 DM-6316,"Update newinstall.sh to support Python 3","{{newinstall.sh}} currently insists on installing and checking for python 2.7. This needs to be changed to allow Python 3. Requires {{sconsUtils}} works with Python 3 as the {{lsst}} EUPS package is installed as part of {{newinstall.sh}}.",1 DM-6317,"Update developer guide to include Python 3","Update the developer guide to indicate that Python 3 must be supported and that code must run on Python 2.7 and 3. This ticket will reference the tech note delivered as part of DM-6315. Writing extensive user documentation on the {{future}} package is beyond the scope of this ticket.",2 DM-6319,"Port sconsUtils to Python 3","{{sconsUtils}} has to be modified to ensure it works with Python 3. Additionally SWIG calls must be changed to trigger Python 3 mode.",3 DM-6320,"Port utils to Python 3","Ensure that the {{utils}} package will work with Python 3.",2 DM-6322,"Port base package to Python 3","Ensure that {{base}} works with Python 3.",2 DM-6323,"Lead Python 3 migration at All Hands Meeting","* Prepare for all hands meeting. * Present plan to developers. * Advise developers doing migration. * Contribute fixes as required.",8 DM-6325,"Replace BOOST_STATIC_ASSERT with static_assert","Replace BOOST_STATIC_ASSERT with static_assert from C++11.",1 DM-6326,"reST roles for mock code references","Add mock code reference roles so that authors can add semantics to their writing without attempting to make actual references to API documentation that does not _yet_ exist. Covers all roles in the Python domain, and supports tilde syntax for collapsing the namespace.",1 DM-6331,"Shifting F16 milestones to S16","Per [~jbecla]'s request, provide Kevin with a list of milestones which we will not address in F16. Reschedule them to S16 in JIRA.",1 DM-6343,"Update LDM-151 introduction to reflect new structure","Some proposals made on DM-6247 to change the structure of LDM-151 (add Algorithmic Components section, move overview into production-specific sections, add notation section to introduction) were accepted at the live meeting on 6/27. This issue rewrites the introduction accordingly.",1 DM-6345,"Firefly Python API scope and decision","Python API to use Firefly visualization components and other functions. This story is to come up with a good plan for the rest of the development work to related to Python API. ",2 DM-6346,"User installation and operation instructions for conda ","Create documentation for the Stack conda binaries created in DM-5415 as part of the Science Pipelines documentation",3 DM-6347,"Add FlagDecorator to support FlagHandler in Python","DM-4009 added the C++ and swig changes needed to allow the FlagHandler to be used from Python. During review, Nate suggested that a decorator class could be used to improve the use of this code in Python. This ticket will be to review Nate's decorator and confirm that it is the correct model for Python-only plugins. We will also modify the unit test in DM-4009 and the EmPsfApprox plugin in DM-6123 to use the decorator.",2 DM-6348,"Write Calibration Products Production section of LDM-151","Write photometric calibration pipeline section of LDM-151",20 DM-6349,"Replace cameraGeom PAF files","PAF files have long been deprecated, but continue to be used for describing the camera geometry. We need to replace the PAF cameraGeom files used for CFHT-MegaCam, DECam, LSSTSim and SDSS, and the scripts used to convert these files to FITS files for reading by the Mappers. They might be replaced by a configuration like YAML, or pure python.",10 DM-6350,"Generate camera description at build time","Camera geometry used to be defined using PAF (policy) files, which are now deprecated. As part of the transition to the refactored camera geometry scheme, scripts were introduced to convert from the PAF files to the new camera geometry configuration scheme which uses FITS files and a python file to describe the camera. These scripts are still part of the obs_* packages, and some people rely on them for making changes to the camera description. On the other hand, the generated FITS files and python file are also first-class members of the obs_* packages. This means that we have two sources of the same information, which is dangerous. For obs_lsstSim, obs_decam, obs_cfht and obs_sdss, we want these scripts to be the primary source of information. This means we should delete the generated files, and create them at build time. We should also standardise the name of the script used to generate these.",3 DM-6351,"Add skeleton words to LDM-151 for AP","We need to flesh out the algorithmic components and narrative sections to the point of having ~1 sentence per paragraph in the finished document.",10 DM-6352,"Use the HTM based reference catalogs in tests","In order to move A.net out of meas_astrom to make it a true dependency, we need to replace its use in tests.",4 DM-6356,"Add linearity correction to obs_decam","Add linearity correction from DM-5462 to obs_decam using the standard linearity tables. ",4 DM-6357,"Take part in LDM-151 Progress Meeting, 2016-06-03",NULL,1 DM-6358,"Take part in LDM-151 Progress Meeting, 2016-06-03",NULL,1 DM-6359,"Take part in LDM-151 Progress Meeting, 2016-06-03",NULL,1 DM-6360,"Update ""Using Boost"" section in DM Developer Guide to prefer standard library by default","Implement RFC-185 by updating the ""Using Boost"" section in DM Developer Guide to prefer standard library by default.",1 DM-6361,"Replan (June)",NULL,11 DM-6362,"Replan (July)",NULL,11 DM-6363,"Replan (August)",NULL,11 DM-6364,"Design DAX containers",NULL,6 DM-6365,"L1 DB Prototype (June)","Placeholder for L1 database prototyping in June 16 -- to be replaced with actual stories of same sp total",14 DM-6368,"Adjust version check of EUPS python package to allow v3","To enable Python 3 support of the stack the EUPS {{python}} stub package needs to allow Python 3. ",1 DM-6369,"Test DIA simulation script with Postgres","I will be useful to compare MySQL and Postgres performance for use in L1. After DM-6918 is complete (means works with MySQL) verify that it can also run against Postgres. ",5 DM-6370,"L1 DB Prototype (August)","Placeholder story for work in this epic in August -- replace with detailed stories at same sp load",14 DM-6373,"Improve skeleton for LDM-151 Algorithmic Components","For all subsections in Algorithms Components owned by [~jbosch]: - Provide enough bullet points to capture scope. - Add bullet points for subtly difficult aspects of components. - Add extra level subsubsubsection level for Measurement. - Create matrix of measurement algorithms and contexts. ",1 DM-6375,"New image visualization functions (F16)","TO support the pipeline QA and build the first web portal, there will be new functions need to be developed. This epic is to collect those functions.",40 DM-6376,"Implement DAX containers","Implement containerized DAX services",4 DM-6377,"SPIE conference 2016","Activity related to attending the SPIE meeting in June 2016. Deliverable is a report on the conference.",15 DM-6378,"Persist output of simple DCR correction","DM-5695 will create transfer matrices stored as numpy arrays. This ticket extends that work to determine a useful format and write functionality to persist those arrays.",2 DM-6379,"Investigate CAOM","Investigate observation model interfaces and storage, and applicability of CAOM",4 DM-6380,"Attend SPIE conference","This ticket relates to attendance at the SPIE meeting in Edinburgh the last week of June.",10 DM-6381,"ADQL support in dbserv","Work on understanding coordinate systems in ADQL and implement the ADQL->qserv rewriter",6 DM-6382,"Generate template DCR images","DM-5695 will create transfer matrices that can be used to create template images of a field at arbitrary airmass. This ticket is to write the code to generate and persist those template images.",4 DM-6383,"Use template DCR images for image differencing","DM-6382 creates template images of a field at arbitrary airmasses, which can be used to match the template airmass to the science image precisely to mitigate Differential Chromatic Refraction in image differencing. This ticket is to determine the best method to supply the new templates to image differencing, which may be simply to create a new exposure and ingest/process the template as though it were a real observation.",2 DM-6384,"Create and deploy Conda binaries for v12.0 release build","Deploy Mac OS X and CentOS 5 conda binaries for v12 to a conda repository (http://conda.lsst.codes/stack/current).",21 DM-6385,"Create CLI tool to add mac users.","Create a CLI script to add and delete Mac OS X users. Somehow this is a many step process on Mac OS X.",1 DM-6387,"Deploy Conda repository to S3","Deploy conda binaries to s3 using their static website feature. http://conda.lsst.codes/stack/current.",4 DM-6388,"Create Ansible automation to run the conda build","Create an Ansible deploy to automate Conda binary builds. Target Mac OS X and CentOS5.",5 DM-6389,"Create CentOS5 Conda binaries","Create CentOS5 conda binary builds using docker then push them to the S3 static website.",2 DM-6392,"Text on variability characterization for LDM-151","Expand the variability characterization algorithmic section of LDM-151.",1 DM-6394,"DM Replanning: ConOps Development","Development of Concept of Operations documents for various DM services, including Data Backbone, AA system, L3 Hosting, and Batch Processing for the commissioning phase. Deliverable: ConOps documents Staff: Don Petravick, Margaret Johnson, Jason Alt, Steve Petrowicz, Hsin-Fang Chiang, Jagadeesh Yedetore, Jim Basney, Alex Withers, Robert Gruendl (roughly 0.5 effort each) Effort: 33 days Planned Start: 6/6/2016 Planned End: 8/31/2016",66 DM-6395,"Data Backbone conops iteration 1: create raw draft (internal)","Write a raw draft of the concept of operations for data backbone services. In this iteration the document is developed in Google docs following the ConOps template.",3 DM-6396,"Data Backbone conops iteration 2: group review to produce first draft","Review raw draft of concept of operations for the data backbone services to work through underdeveloped areas, clear up uncertainties, and make readable.",2 DM-6397,"Data Backbone conops iteration 3: larger review to produce second draft","Review first draft of data backbone services conops within Data Processing Architecture working group, bringing in relevant experts. Input from review is incorporated into a second draft.",6 DM-6398,"Data Backbone conops formatting: convert second draft to reStructuredText","When the data backbone services conops is in a solid state, convert the Google doc to reStructuredText following DM's documentation versioning process.",2 DM-6399,"DM replanning: NCSA WBS broken down to Level 3 WBS elements","Break down NCSA restructured WBS to level 3 (02C.07.xx.xx) sufficient for integration into PMCS. Deliverable: Draft WBS for project-level change processes to accommodate. Staff: Don Petravick, Margaret Johnson, Jagadeesh Yedetore, Santanu Chaudhuri (roughly 0.5 each) Effort: 5 days Planned Start: 6/1/2016 Planned End: 6/30/2016",10 DM-6400,"DM replanning: phased WBS elements for minimal data archiving of camera data and minimal transport via data backbone to NCSA","Phased breakdown of work activities to construct minimal archiving of camera data and transport via data backbone to NCSA. Includes release of claim on camera buffer. Deliverable: Phased WBS for minimal data archiving of camera data and minimal transport via data backbone to NCSA. Staff: Don Petravick, Margaret Johnson, James Parsons, Steve Petrowicz, Jagadeesh Yedetore, Felipe Menanteau Effort: 20 days Planned Start: 7/1/2016 Planned End: 7/31/2016",40 DM-6401,"DM replanning: instantiating Project Reporting Group and planning & reporting process","Instantiating NCSA's Project Reporting Group and planning and reporting processes. Deliverable: Project reporting normalized to new methods. Staff: Don Petravick, Margaret Johnson, Santanu Chaudhuri, Jagadeesh Yedetore Effort: 10 days Planned Start: 6/6/2016 Planned End: 6/30/2016",20 DM-6403,"DM replanning: participation in PM working group","Participation in Project Management working group for DM replanning. Deliverable: Deliverables to PM working group. Staff: Santanu Chaudhuri, Don Petravick, Margaret Johnson Effort: 3 days Planned Start: 6/1/2016 Planned End: 6/30/2016",6 DM-6404,"Operations Planning in LOPT and TOWG","Participation in LOPT and TOWG for LSST operations planning to get to ConOps. Deliverable: Deliverables to get to ConOps Staff: Don Petravick, Margaret Johnson Effort: 12 days Planned Start: 6/6/2016 Planned End: 8/31/2016",24 DM-6405,"Verification Planning with Systems Engineering","Extend concepts of operations to include adequate verification. Will involve coordination with LSST Systems Engineering team. Deliverable: verification plan descriptions in ConOps documents Staff: Don Petravick, Margaret Johnson, Jason Alt, Paul Wefel, Steve Petrowicz, Jagadeesh Yedetore Effort: 27 days Planned Start: 6/6/2016 Planned End: 8/31/2016",54 DM-6406,"Install and configure GNU/Linux on my office desktop.",NULL,4 DM-6407,"Main prototype for all L1 entities (F16 part 1)","Main program prototypes for all entities in the L1 Prompt Processing system. This epic continues work started in x16 and covers work in the first part of the F16 cycle. Deliverable: Major component hierarchy and all 'has a' objects list. Staff: Jim Parsons + 2 summer students (at 50% each) Effort: 15 days Planned Start: 6/1/2016 Planned End: 7/31/2016",30 DM-6408,"L1 Startup Scaffolding","Scaffolding to remotely start and stop all L1 Prompt Processing system entities. Deliverable: Initial scripts starting and stopping all entities remotely Staff: Jim Parsons + 2 students (50% each) Effort: 8 days Planned Start: 7/1/2016 Planned End: 7/31/2016",16 DM-6409,"L1 System Status Message Dictionary","Message formats and contents between all L1 Prompt Processing System entities. This epic continues work from x16. Deliverable: Enumeration of message formats and contents as needed. Staff: Jim Parsons + 2 students (50% each) Effort: 10 days Planned Start: 6/1/2016 Planned End: 6/30/2016",20 DM-6410,"Learn about design principles and usage of data processing tasks.","As we are working on designing a new workflow framework I should read available documentation and look through the existing code base regarding data processing tasks to learn more how they are written, work, and interact among themselves.",10 DM-6411,"Message interaction between all L1 entities (F16)","Basic messaging and interactions, including data dictionary and message patterns. Deliverable: All communication, data, and reporting paths specified and implemented Staff: Jim Parsons + 2 students (50% each) Effort: 15 days Planned Start: 6/1/2016 Planned End: 8/31/2016",30 DM-6412,"Basic Framework for L1 System Health and Status","Basic framework for L1 prompt processing and archiving system health and status display, including status event recorder. Deliverable: First draft of health checks via message; plan for remote diagnostics Staff: Jim Parsons + 2 students (at 50% each) Effort: 10 days Planned Start: 7/1/2016 Planned End: 8/31/2016",20 DM-6414,"ConOps/Planning Document for the OCS-DM interface","Detailed concept and technical implementation plan for DM interface with OCS. Deliverable: ConOps document for OCS Staff: Jim Parsons Effort: 7 days Planned Start: 8/1/2016 Planned End: 8/31/2016",14 DM-6415,"ConOps/Planning Document for the Camera-DM interface","Detailed concept and technical implementation plan for DM interface with the main Camera. Deliverable: ConOps document for Camera Staff: Jim Parsons Effort: 7 days Planned Start: 8/1/2016 Planned End: 8/31/2016",14 DM-6416,"ConOps/Planning Document for Base Archiving API","Detailed concept and technical implementation plan for Base Archiving API. Deliverable: ConOps document for Base Archiving API Staff: Jim Parsons Effort: 5 days Planned Start: 8/1/2016 Planned End: 8/31/2016",10 DM-6417,"Specification for Comfort Dashboard and Alarms","Detailed specifications for comfort dashboard and alarms, including interactions with human operations, and some technology prototyping. Deliverable: Detailed specifications Staff: Jagadeesh Yedetore, Santanu Chaudhuri, TBH Effort: 21 days Planned Start: 6/13/2016 Planned End: 8/31/2016",42 DM-6418,"Final version of Supertask and Activator prototype implementation","Final version of Supertask and Activator prototype implementation. This is not the final version of SuperTask, just what was initially designed. Deliverable: Final prototype Staff: Matias Carrasco-Kind, Mikolaj Kowalik Effort: 6 days Planned Start: 6/6/2016 Planned End: 7/15/2016",12 DM-6419,"Propose and discuss workflow selection","Propose and discuss workflow selection. Deliverable: RFC for workflow selection Staff: Hsin-Fang Chiang, Mikolaj Kowalik, Rob Kooper, Steve Pietrowicz Effort: 5 days Planned Start: 7/1/2016 Planned End: 7/15/2016",10 DM-6420,"Investigate Shifter, HTCondor, preemption, and file cleanup","Evaluate use of containers to configure HTCondor slots for DRP tasks. Investigating Shifter on Blue Waters. Deliverable: investigation report Staff: Greg Daues Effort: 10 days Planned Start: 6/1/2016 Planned End: 8/31/2016",20 DM-6421,"Evaluate use of dynamic slot mechanism in HTCondor for DRP tasks","Evaluate use of dynamic slot mechanism in HTCondor for DRP tasks. Deliverable: Evaluation report Staff: Steve Pietrowicz Effort: 5 days Planned Start: 6/1/2016 Planned End: 6/30/2016",10 DM-6422,"Python 3 migration work","Migrate existing packages in the LSST stack to support Python 3 compatibility. Some work will occur during the All-Hands meeting. Deliverable: Migration of existing packages for Python 3 compatibility Staff: Steve Petrowicz, Jim Parsons, Matias Carrasco-Kind, Mikolaj Kowalik, Hsin-Fang Chiang Effort: 1 days Planned Start: 8/1/2016 Planned End: 8/31/2016",2 DM-6423,"Data Backbone: explore overheads and costs of staging model","Explore overheads and costs of staging model from Data Backbone into data caches. Deliverable: assessment and characterization of staging component of orchestration Staff: Steve Pietrowicz Effort: 15 days Planned Start: 8/1/2016 Planned End: 8/31/2016",30 DM-6425,"Analyze existing implementation of Supertask","Look through the code base of current prototype of Supertask and Activator to understand better limits of existing design of data processing task and how they are being addressed.",6 DM-6426,"Understand how EUPS works","Read Developer's Guide tutorial and official documentation of EUPS to understand how it works to manage Stack's packages easily.",2 DM-6427,"Data Backbone: produce abstract API to ingest data into L1 archive ","Implement an abstract API to ingest data into the L1 archive in the Data Backbone. Deliverable: abstract API in github Staff: Steve Pietrowicz Effort: 4 days Planned Start: 6/1/2016 Planned End: 6/30/2016",8 DM-6428,"Data Backbone: initial analysis of OpenStack and Ceph object store APIs","Initial analysis of OpenStack and Ceph object store APIs in Data Backbone. Deliverable: analysis report Staff: Steve Pietrowicz Effort: 10 days Planned Start: 6/1/2016 Planned End: 6/30/2016",20 DM-6429,"Data Backbone: initial analysis of DDN WOS object store API","Initial analysis of DDN WOS object store API in Data Backbone. Deliverable: analysis report Staff: Steve Pietrowicz Effort: 10 days Planned Start: 7/1/2016 Planned End: 7/31/2016",20 DM-6430,"Object Store and WAN Data Interchange Evaluation","Evaluate data interchange over the WAN with object store technology. Deliverable: evaluation report Staff: Jason Alt, TBD SET hire Effort: 25 days Planned Start: 6/6/2016 Planned End: 8/31/2016",50 DM-6431,"Commission the evaluation framework on one machine at NCSA","Commission the evaluation framework on one machine at NCSA.",10 DM-6432,"Evaluate at additional sites and/or by simulation using WAN emulation techniques","Additional sites and/or simulation using WAN emulation techniques.",30 DM-6433,"Produce evaluation report","WAN data interchange evaluation report.",5 DM-6434,"Cost Model Updates for FY17","Semi-annual updates of costing forecast in sizing model. Deliverable: RFC for proposed cost updates Staff: Jason Alt Effort: 10 days Planned Start: 6/6/2016 Planned End: 6/30/2016",20 DM-6435,"Service Management for F16 (part 1)","Provide service management for LSST development resources at NCSA, including Nebula openstack cluster, lsst-dev, and relevant FY16 procurements. Interface with functional groups to provide support for DM services. Deliverable: services Staff: Greg Daues Effort: 6 days Planned Start: 6/1/2016 Planned End: 8/31/2016",12 DM-6436,"Deploy FY16 Integration Environment","Deploy infrastructure for FY16 SUI/Qserv integration environment (a.k.a., prototype DAC). Deliverable: secure SUI/Qserv integration environment Staff: 3 NCSA ICI engineers (networking, storage, systems) Effort: 65 days Planned Start: 6/1/2016 Planned End: 6/30/2016",100 DM-6437,"Convert GWT projection and Coorindate Conversion routine to JavaScript","convert Booth's projection code and Judy Bennet's coordinate conversion routines to pure javascript ",8 DM-6438,"Test the new JS convertion and projection routines against the java versions","Now that the gwt algorithmic code has been converted it needs to be validated. Set up the test for both the projections routines and the coordinate conversion routines. h3. Task Details * Unit test should be run on the Java and JavaScript side * A unit test should exist for all 10 projections. More might be necessary since there are variation within a projection type. * We need to bring Booth in for details of each projection and each variation. * Booth has example file somewhere. * Xiuqin ran coordinate conversion test in to past. She (and maybe Booth) have the best understand of what that test should be. * The same input file should be used for the Java and JavaScript side. * The same output (if possible) should be produced. h3. Status so far *Projections* * GNOMONIC - somewhat tested * ORTHOGRAPHIC -somewhat tested * AITOFF - somewhat tested * SFL - somewhat tested * PLATE - somewhat tested * LINEAR * ARC * CAR * NCP * CEA *Coordinate conversion.* It appears to work when run in Firefly, however the code is not fully covered. h3. Switching between GWT and pure JavaScript To switch between GWT and JS edit {{VisUtil.js}} and change the following line: {{export const USE_GWT= false;}} true uses the old code, false used the new code. h3. Entry Points and Directories * Coordinate Conversion: _Dir_: firefly/src/firefly/js/astro/conv, _Entry Point:_ VisUtil.js, convert(), line 104, also see line 26 * Projection: _Dir_: firefly/src/firefly/js/visualize/projection, _Entry Point:_ WebPlot.js, makeWebPlot(), line 124 ",12 DM-6439,"Remove GWT from build","After the code is tested remove the GWT from the build. Should check with [~roby] to make sure the boolean to enable GWT has been removed from VisUtil.js and WebPlot.js",4 DM-6440,"May 2016 LAAIM work","Gave input on IAM design for FY16 Integration Environment. Discussed IAM replication requirements with stakeholders. Attended local NCSA LSST coordination meeting.",4 DM-6441,"Create Ansible automation to run the conda build","Complete Ansible implementation started in DM-6388.",4 DM-6443,"Measure photometric and astrometric precision for DECam COSMOS dataset","Measure the photometric and astrometric precision for the DECam COSMOS dataset and determine the sources of extra systematic scatter.",26 DM-6444,"For the 3_build-git-image.sh, pass -j$(nproc) to scons to speed up the build process",NULL,2 DM-6445,"Verification CoDR preparation","Just capturing FE's SPs towards this. ",6 DM-6446,"Remove boost dependencies where possible","In X16/DM-5580, we removed Boost from a number of packages. However, we may not have rigorously updated their dependency lists to indicate where Boost is no longer required. Please do so.",1 DM-6447,"Revise and improve DMTN-020","An initial version of DMTN-020, describing project management practices, was produce in DM-6140. Revise and update that based on feedback from the DM Project Manager, DM Project Controls Specialist, DM technical managers, and others.",10 DM-6448,"Deploy FY16 Storage Expansion (part 2)","Deploy infrastructure for FY16 storage expansion. This epic covers follow-on work to DM-3830. Deliverable: storage expansion Staff: 5 NCSA ICI engineers (networking, storage, systems) Effort: 45 days Planned Start: 6/1/2016 Planned End: 7/31/2016 ",90 DM-6449,"Deploy FY16 Nebula Expansion (part 2)","Deploy infrastructure for FY16 Nebula expansion. This epic covers follow-on work to DM-3832. Deliverable: expanded services Staff: 3 NCSA ICI engineers (networking, storage, systems) Effort: 10 days Planned Start: 6/1/2016 Planned End: 7/31/2016",20 DM-6450,"Deploy FY16 Cluster Services (part 2)","Deploy infrastructure for FY16 cluster services. This epic covers follow-on work to DM-5624. Deliverable: cluster services deployed Staff: 3 NCSA ICI engineers (networking, storage, systems) Effort: 10 days Planned Start: 6/1/2016 Planned End: 6/30/2016 ",20 DM-6451,"Deploy FY16 Verification Cluster (part 2)","Deploy infrastructure for FY16 Verification Cluster. This epic covers follow-on work to DM-5626. Deliverable: verification cluster Staff: 5 NCSA ICI engineers (networking, storage, systems) Effort: 65 days Planned Start: 6/1/2016 Planned End: 6/30/2016 ",100 DM-6452,"L1 System Mock 1: Butler component","Implement a mock program that receives the Level 1 processing system data stream to simulate Butler integration. Deliverable: mock program Staff: Felipe Menanteau Effort: 5 days Planned Start: 7/1/2016 Planned End: 7/31/2016",10 DM-6453,"L1 System Mock 2: Archive component","Integrate mock API to ingest data into Archive that organize data spatially on tape. Deliverable: mock API Staff: Felipe Menanteau Effort: 5 days Planned Start: 8/1/2016 Planned End: 8/31/2016",10 DM-6454,"Investigate the feasibilty of hosting an extracted, transformed and loaded database at archive site instead of a full EFD","Investigate the feasibilty of hosting an extracted, transformed and loaded (ETL) database at archive site instead of a full EFD. Gather sufficient details to support change request to eliminate full EFD at archive site. Address evident concerns relating to the increased data volume in the reformatted EFD and better integration into operational context, including the data backbone. Deliverable: sufficient details to support a change request Staff: Steve Peckins Effort: 16 days Planned Start: 7/1/2016 Planned End: 8/31/2016 ",32 DM-6455,"Investigation of workflow and interface tools in the OpenStack environment","Investigation of workflow and interface tools in the OpenStack environment. This is learning to support the eventual toolkits anticipated in the SUI for production deployment at the DAC at NCSA. Deliverable: investigation report Staff: Matias Carrasco-Kind, 2 students (at 50% each) Effort: 20 days Planned Start: 6/15/2016 Planned End: 8/15/2016",40 DM-6456,"create description of features in storage APIs","The APIs for the storage brokers we're looking into are similar, but don't have a 1-1 correspondence. Write up the features offered by the APIs, and see where there is overlap.",4 DM-6457,"Design and RFC for Repository Refactor","Drive the RFC for Repo Refactor to completion (this includes a lot of design work)",20 DM-6458,"Expand skeleton in LDM-151","We need to flesh out the skeleton text to contain full descriptions of the algorithms and pipelines we expect the baseline design to use.",20 DM-6459,"productize ""Repository Refactor""","After RFC-184 is closed: implement, unit tests, review, document, submit. When this story closes, I think RFC-184 status is supposed to be changed from Adopted to Implemented.",20 DM-6460,"Ramp-up adminstrative capability of qserv for deployment of SUI prototype system","New staff will be on-boarded who has no prior experience with qserv. The goal is to ramp up to provide administration for the deployment of the prototype DAC, and to foster development of documentation within the qserv project that facilitates administration and usability of the product as a component in the LSST systems. Deliverable: administration of qserv for SUI prototype system Staff: Steve Peckins Effort: 15 days Planned Start: 6/1/2016 Planned End: 8/31/2016",30 DM-6461,"App logging framework migration work","App logging framework migration work. Enhance the lsst::log package, prepare and do a RFC, migrate codes to use lsst::log, deprecate pex_logging. Deliverable: framework migration Staff: Hsin-Fang Chiang Effort: 20 days Planned Start: 7/1/2016 Planned End: 8/31/2016",40 DM-6462,"Emergent middleware work (F16, part 1)","Reserve of effort to handle MINOR middleware-related work that emerges during the F16 cycle, June-August. Deliverable: TBD middleware fixes Staff: Steve Pietrowicz, Hsin-Fang Chiang, Mikolaj Kowalik, Matias Carrasco-Kind, Rob Kooper Effort: 10 days Planned Start: 6/6/2016 Planned End: 8/31/2016",20 DM-6463,"Please provide how-to-reproduce instructions for LSST/HSC comparison epics","For all the stories describing comparisons between HSC and LSST results (notably DM-5301 and DM-5827), please provide instructions describing the steps to reproduce the comparison. In particular, include: * A list of any tweaks that had to be applied to the code; * Non-default configuration options; * Exactly which comparisons were made.",2 DM-6464,"Compare CModel results from LSST and HSC","Demonstrate that the HSC and LSST stacks produce consistent results for CModel measurement. Account for (and fix, where relevant) differences.",4 DM-6465,"Compare Kron results from LSST and HSC","Demonstrate that the HSC and LSST stacks produce consistent results for Kron measurement. Account for (and fix, if relevant) any differences.",4 DM-6466,"Compare meas_mosaic-ed HSC and LSST coadds","The previous comparison of coadds on HSC and LSST was performed without an operable LSST-based meas_mosaic. When one becomes available, demonstrate that mosaicking is consistent between HSC and LSST; describe, account for, and (where possible) correct differences.",5 DM-6467,"Account for noise replacement differences between LSST and HSC","In DM-5827, [~rearmstr] wrote: {quote} In most of these plots you can see some scatter at relatively bright magnitudes... These are likely getting different pixel values when we replace objects with noise which is causing these changes. {quote} Check that this is the case, and, if so, explain why the noise replacement is different.",5 DM-6470,"convert jenkins-ebs-snapshot job to use credentials for aws keys","At present, this job is being templated by puppet to inject the keys in plain text which are they converted by jenkins to stored secrets if/when the job is edited and resaved via the jenkins UI. This means that the credentials may be leaked.",1 DM-6471,"Conda eups packages don't work if eups is already configured","When [~tjenness] attempted to use the Conda repository he ran into a problem installing and using the packages because he already had an active EUPS_DIR and EUPS_PATH. When the eups package is installed and linked, it should warn users when these environment variables are set. I'm open to another solution but would prefer it doesn't change the eups package behavior. Changing behavior goes against the best practices for Conda and more generally packaging.",1 DM-6473,"Possible image related issues in firefly viewer","Image Meta Data tab * images cannot be remove, but in expanded mode, it can. * selecting image no longer highlight table. the reverse works fine. * visualize/saga/ImageMetaDataWatcher.js:272 returns -1. * when a non-meta table is selected, images are shown, but not the toolbar. * after table is removed, images are still there. image external api does not mix well with firefly viewer. * firefly viewer uses 'triViewImages’ viewer_id while api has no viewer_id. as a result, images loaded by api will be lost once table or other searched data are returned. catalog overlay are drawn outside of the images. more issues: - It's possible to select distance tool and then area selection. First drag would define area selection, all the following line. A click would be defining a 0 length line, even if point selection is enabled. ",8 DM-6474,"Restore star selector registry","Restore the registry for star selectors that was lost in DM-5532, now that tasks in registries can be used as subtasks. Also use the registry where appropriate.",2 DM-6475,"Install conda psutil instead of LSST's version","[~tjenness] requests that conda-lsst uses Conda's psutil. Currently we use our own version.",1 DM-6476,"Report and work around conda repository change","This needs to be worked around by either using a different version of conda-build or addressing the changes to the conda recipe structure. I also want to comment and/or create an issue on conda-build so they know that such changes are affecting users. See: https://github.com/conda/conda-build/issues/1003 https://github.com/conda/conda-build/pull/1004 https://github.com/conda/conda-build/commit/b4ec0e0659d8f376042d4fc391616bf235996cf5 [Mario fixed it|https://github.com/mjuric/conda-lsst/commit/6a552b6f9cada2530681cfdc4a9f67add261ff99] but that fix will be broken as soon as conda-build #1004 is merged.",1 DM-6487,"Form validation regression issues","Recent changes in 'dev' made some of the components stop working. We need to click through firefly viewer to identify the problems and fix it. Below are a few that I've spotted. Data Sets menu: Form fail validation when they should not. filters and upload file should be nullable. Catalogs Classic menu: form fail validation without any visual indications. valid parameter is false when passed into validUpdate in CompleteButton. xyPlot(Scatter Plot) options: Beside X and Y, everything else should be optional(nullable). Even after entering a valid value into all of the fields, 'OK' still fail validation similar to above where 'valid' parameter is false when passed into CompleteButton. There may be more, please do a quick search to make sure all usage of CompleteButton is good. ",4 DM-6490,"Investigate calibration zeropoint offset between HSC vs. LSST processCcd.py runs","As reported in DM-4730, while the scatter between single frame processing measurements of the same dataset on the HSC vs. LSST stacks is quite good (rms = 0.009 mag between Gaussian fluxes, for example, in the figure shown on that ticket), there is a clear offset (0.0166 mag in the figure shown) in the zeropoint between the two stacks (it is systematic, i.e. no trend with magnitude). The cause may well be due to slight differences in the reference stars selected for calibration. We also speculated about differences in slot definitions used in the calibrations steps (e.g. for aperture corrections, psfex, etc...), so I have rerun visit 1322 through both stacks having forced all apertures used in calibration to be the same, namely a circular aperture of 12 pixels measured using the sinc algorithm (as opposed to ""naive""). I have attached the *processCcd.py* config files for the two runs so my settings can be reproduced. Also of note, I am using a {{meas_algorithms}} branch on the HSC stack with the following commit: {code} commit 173ad0b32ed4f4ab074f1a942d2d3f758e189917 Author: Lauren MacArthur <lauren@astro.princeton.edu> Date: Wed Jan 13 16:35:59 2016 -0500 Hack to allow flux.aperture to be used in apCorr Since it does not seem possible to access the nth element of a schema element that is an array in the context of setting a config override, this allows for flux.aperture to be set as calibrate.measureApCorr.reference and it sets it to index 4 (which corresponds to a radius of 12 pixels) in the __init__. This was selected to match the current LSST default. diff --git a/python/lsst/meas/algorithms/measureApCorr.py b/python/lsst/meas/algorithms/measureApCorr.py index 9f6c599..f1fa99d 100644 --- a/python/lsst/meas/algorithms/measureApCorr.py +++ b/python/lsst/meas/algorithms/measureApCorr.py @@ -81,6 +81,9 @@ class MeasureApCorrTask(lsst.pipe.base.Task): def __init__(self, schema, **kwds): lsst.pipe.base.Task.__init__(self, **kwds) self.reference = KeyTuple(self.config.reference, schema) + if self.config.reference == 'flux.aperture': + print ""NOTE: setting aperture correction flux to flux.aperture[4] ==> radius = 12 pixels"" + self.reference.flux = self.reference.flux[4] self.toCorrect = {} {code} I attach some of the figures comparing the PSF fluxes from these runs which compare the output of the two stacks having matched the two src catalogs. There are two sets: 1) having adjusted the flux for each source to the zeropoint calculated in the calibration and stored as *FLUXMAG0* 2) having adjusted the flux for all sources to a common zeropoint (zp=33.0, chosen to roughly match the calibrated zp). Note that my figures do include aperture corrections (in DM-5301, many of the plots show fluxes pre-aperture correction). I have also included plots that directly compare the aperture corrections applied (difference in mag units). Finally, I also include plots comparing the 12 pixel circular aperture mags (i.e. to which no apCorr is added). Clearly, the zeropoint determined in the calibration of the two stacks differs between the two stacks and, in particular, there seem to be some very problematic CCDs where the differences are particularly significant (~0.05 mag, and not always in the same direction). Please investigate the source of this discrepancy.",8 DM-6491,"Investigate offset in baseline zeropoint between LSST vs. HSC stack reductions for some HSC visits","DM-6490 reports on an offset between the calibration zeropoints between HSC vs. LSST *processCcd.py* runs. Here we report another, additional, offset seen in certain HSC visits. It is not seen in the figures shown in DM-6490 for visit 1322. However, here attach the same figures for visit 19696, run with identical setups/configs for both stacks as in DM-6490, where we see an additional offset in the ""common ZP"" figures (i.e. all fluxes have been scaled the same zp=33.0 for comparison). A best guess at present is that the calibration frames are different between the HSC and LSST stacks for the timeframe of this visit; e.g. were the inputs ingested exactly the same for both sets? Did the bug in regards to flagging on the flats noted in DM-5124: {quote} I found a difference in the codes doing the statistics: the HSC code uses a hard-coded mask ignore list of DETECTED only, while the LSST code uses a configurable mask ignore list that defaults to DETECTED,BAD (and the default isn't overridden). This produces a large difference on CCDs with bad amps (e.g., ccd=9). There's a smaller difference on ccd=49 because the number of BAD pixels is smaller. Also note that the scaling of one CCD (like ccd=9) can affect others because we force the normalisations to correspond to that which we get from solving the system of M exposures of N CCDs. {quote} have a greater impact on these calibs? Please investigate the cause of this offset.",8 DM-6492,"Review LTS-210",NULL,2 DM-6494,"Better error messages from the camera mapper when a template cannot be formatted","The CameraMapper produces a very unhelpful traceback if it cannot format a template string with the provided data ID dict. For example: {code} Traceback (most recent call last): File ""/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/pipe_tasks/12.0.rc1-3-gb785bf9/bin/processCcd.py"", line 25, in <module> ProcessCcdTask.parseAndRun() File ""/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/pipe_base/12.0.rc1-1-g832266b/python/lsst/pipe/base/cmdLineTask.py"", line 450, in parseAndRun parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides) File ""/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/pipe_base/12.0.rc1-1-g832266b/python/lsst/pipe/base/argumentParser.py"", line 479, in parse_args self._processDataIds(namespace) File ""/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/pipe_base/12.0.rc1-1-g832266b/python/lsst/pipe/base/argumentParser.py"", line 577, in _processDataIds dataIdContainer.makeDataRefList(namespace) File ""/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/pipe_base/12.0.rc1-1-g832266b/python/lsst/pipe/base/argumentParser.py"", line 126, in makeDataRefList dataRef=dr)] File ""/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/pipe_base/12.0.rc1-1-g832266b/python/lsst/pipe/base/argumentParser.py"", line 935, in dataExists return butler.datasetExists(datasetType = datasetType, dataId = dataRef.dataId) File ""/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_persistence/12.0.rc1-1-gc553c11+4/python/lsst/daf/persistence/butler.py"", line 288, in datasetExists locations = self.repository.map(datasetType, dataId) File ""/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_persistence/12.0.rc1-1-gc553c11+4/python/lsst/daf/persistence/repository.py"", line 392, in map return self.doParents(Repository.doMap, *args, **kwargs) File ""/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_persistence/12.0.rc1-1-gc553c11+4/python/lsst/daf/persistence/repository.py"", line 325, in doParents res = func(parent, *args, **kwargs) File ""/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_persistence/12.0.rc1-1-gc553c11+4/python/lsst/daf/persistence/repository.py"", line 405, in doMap loc = self._mapper.map(*args, **kwargs) File ""/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_persistence/12.0.rc1-1-gc553c11+4/python/lsst/daf/persistence/mapper.py"", line 169, in map return func(self.validate(dataId), write) File ""/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/12.0.rc1+6/python/lsst/daf/butlerUtils/cameraMapper.py"", line 284, in mapClosure return mapping.map(mapper, dataId, write) File ""/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/12.0.rc1+6/python/lsst/daf/butlerUtils/mapping.py"", line 123, in map path = mapper._mapActualToPath(self.template, actualId) File ""/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/12.0.rc1+6/python/lsst/daf/butlerUtils/cameraMapper.py"", line 732, in _mapActualToPath return template % self._transformId(actualId) TypeError: %d format: a number is required, not NoneType {code} It is unclear what string was being formatted with what data, making the problem difficult to diagnose and correct. I suggest changing line 732 of CameraMapper.py from: {code} return template % self._transformId(actualId) {code} to something like the following: {code} try: transformedId = self._transformId(actualId) return template % transformedId except Exception as e: raise RuntimeError(""Failed to format %r with data %r: %s"" % (template, transformedId, e)) {code} Here are the last few lines of the same traceback after applying this change: {code} path = mapper._mapActualToPath(self.template, actualId) File ""/Users/rowen/UW/LSST/lsstsw/stack/DarwinX86/daf_butlerUtils/12.0.rc1+6/python/lsst/daf/butlerUtils/cameraMapper.py"", line 735, in _mapActualToPath raise RuntimeError(""Failed to format %r with data %r: %s"" % (template, transformedId, e)) RuntimeError: Failed to format '%(date)s/%(filter)s/decam%(visit)07d.fits.fz[%(hdu)d]' with data {'date': '2013-02-10', 'ccdnum': 10, 'hdu': None, 'visit': 176837, 'filter': 'z'}: %d format: a number is required, not NoneType {code} A bit wordy, but it is much easier to figure out what went wrong. I have stumbled across this problem twice in the last few weeks, so I consider this change fairly important. The first time it was caused by a defective format string in a paf file. This time I'm not yet sure what is causing it, but at least I have something to go on.",1 DM-6497,"Assist IN2P3 engineer in loading DC2013 data sample","Bogdan Vulpescu, IN2P3 engineer, tried to load DC2013 data sample. Fabrice help was required to install Qserv in multi-nodes and understand data-loading system. Some issues have been found and will be reported in future tickets: - a script to publish loaded data (i.e. insert db name in qservw_worker.Dbs) would be useful - mysql client might break proxy if option are not provided correctly (a bug report will be available soon)",3 DM-6498,"Assist IN2P3 student in using Openstack and following LSST coding standards","[~oachbal] has written a code to automate Qserv cluster boot on Openstack cloud. Soma support was required to understand and solve cloud-init and openstack issues, Qserv container deployment and LSST coding standards.",6 DM-6500,"convert irsaviewer to react.js","Convert irsaviewer to use the new firefly library built on react/flux. also made these changes: remove irsa footer from fireflyviewer filter by selected rows auto-correct table’s filter input. multiple columns sort via sortByCols fix menu item not showing selected ife automatically builds firefly. To test, make sure you pull ife repos as well. same branch name on ife.",10 DM-6501,"Regrid needed to for WebGrid","When compute the points for the grid lines, there is no guarantee that the number of points will all be the same. However, the points can be regrided to ensure all the lines have the same number of points.",6 DM-6502,"setup test framework","Need to decide what to check so we have a consistent testing, requirement is opensource - language - license - maturity - funding - ease of install -- dependencies - OS requirements - ease to create a workflow - ability to execute on clusters/laptop - test with simple LSST workflow - willingness to meet and answer our questions -- open bug reporting site -- speed at which bugs are resolved -- size of community, external collaborators - how big a graph can it support? - is shared filesystem required for data, or can it take care of data transfer - does it support MPI other parallel code? - smart wrt to data available on node (optional) ",2 DM-6503,pegasus,"Review Pegasus Workflow Management System (https://pegasus.isi.edu) against criteria defined in the epic.",6 DM-6504,Swift,"Review [Swift|http://swift-lang.org/main/index.php] scripting language against criteria defined in the epic.",6 DM-6505,"final report",NULL,4 DM-6506,panda,NULL,6 DM-6510,"Verification Plan Systems Engineering Status Review","This follows on from [DM-5315] and covers collating comments from DMLT, submitting the status report and document to Systems Engineering and dealing with the comments. ",4 DM-6513,"Remove unsused ""version.h"" file and associated code","This code seems obsolete and unused: {code:bash} qserv@clrinfopc04:~/src/qserv (tickets/DM-5967)$ grep -r ""version.h"" * admin/tools/docker/git/src/qserv/site_scons/genversion.py:# genversion.py : declare a builder for global version headers. admin/tools/docker/git/src/qserv/site_scons/genversion.py: """"""Construct a version header from git-describe output and store admin/tools/docker/git/src/qserv/core/modules/SConscript:versionFile = env.Command(['global/version.h'], None, genversion.buildVersionHeader) core/modules/SConscript:versionFile = env.Command(['global/version.h'], None, genversion.buildVersionHeader) {code}",3 DM-6514,"Minor fixes to linearization","DM-5462 added linearization to {{IsrTask}} but had a few loose ends which this ticket aims to correct: - I intended to enable linearization by default, but somehow lost that change. - I intended to update obs_test to use null linearization, but I forgot and the previous item meant I didn't catch the omission - It turns out that the butler data proxy object will not work with functors (attempting to call the retrieved item results in an error, rather than resolving the proxy). This is easily worked around by using immediate=True when retrieving linearizers. This didn't show up until DM-6356 because obs_decam is the only camera that uses linearization lookup tables, and obs_subaru avoids the problem by not returning a proxy. ",1 DM-6516,"Convert footprint support","Convert the footprint support from the GWT code",20 DM-6518,"Fix scheduler delays caused by mlock call in memman.","Locking tables in memory with mmap and mlock greatly increases scan query speeds but makes the worker scheduler unresponsive to interactive queries. This also tends to have only one scheduler (fast, slow, medium) running at a given time.",8 DM-6519,"Temp local background broken","The temp local background feature has been broken and needs to be fixed.",1 DM-6520,"Prepare an RFC about logging migration","Summarize RFC-29, evaluate technical details, prepare working examples, re-raise RFC-29 or file a new RFC before the migration. Some implementation may be done before the new RFC. ",10 DM-6521,"Enhance lsst.log by having a Log object and Python interface ","Based on branch u/ktlim/getLogger in {{log}} and requests from DM-3532, implement a lsst::log Python interface through Log objects, and allow controllability of logger names and levels in Python. ",7 DM-6524,"Capture ProjMgmt WG Long Term Planning conclusions in DMTN-020","The ProjMgmt WG is going to agree on a strategy for long term planning. Make sure it's captured in DMTN-020.",3 DM-6527,"Statement of Work ",NULL,10 DM-6528,"Networking Configuration",NULL,5 DM-6529,"Investigate single frame processing astrometry failures/poor solutions on some HSC chip/visits.","The astrometric solution of some visit/ccd combinations for HSC data are failing or finding very poor solutions. This typically occurs for the outermost (highly fringed) ccds (e.g. 100..103, 95). I provide some sample output below. {code:title=LSST bad fit: visit=19696 ccd=100} processCcd.calibrate.astrometry.refObjLoader: Loaded 71 reference objects processCcd.calibrate.astrometry.matcher: filterStars purged 0 reference stars, leaving 71 stars processCcd.calibrate.astrometry.matcher: Purged 4436 unusable sources, leaving 288 usable sources processCcd.calibrate.astrometry.matcher: Matched 6 sources processCcd.calibrate.astrometry.matcher WARNING: Number of matches is smaller than request processCcd.calibrate.astrometry: Matched and fit WCS in 1 iterations; found 6 matches with scatter = 0.000 +- 0.000 arcsec {code} {code:title=HSC fit: visit=19696 ccd=100} 2016-06-10T17:13:54: processCcd.calibrate.astrometry: Found 80 catalog sources 2016-06-10T17:13:54: processCcd.calibrate.astrometry: Matching to 119/148 good input sources 2016-06-10T17:13:55: processCcd.calibrate.astrometry: Matched 20 sources 2016-06-10T17:13:55: processCcd.calibrate.astrometry WARNING: Number of matches is smaller than request 2016-06-10T17:13:55: processCcd.calibrate.astrometry: 20 astrometric matches for 100, 0_31 2016-06-10T17:13:55: processCcd.calibrate.astrometry: Refitting WCS 2016-06-10T17:13:55: processCcd.calibrate.astrometry: Astrometric scatter: 0.038076 arcsec (with non-linear terms, 20 matches, 0 rejected) {code} {code:title=LSST failed fit: visit=19696 ccd=103} processCcd.calibrate.astrometry.refObjLoader: Loaded 68 reference objects processCcd.calibrate.astrometry.matcher: filterStars purged 0 reference stars, leaving 68 stars processCcd.calibrate.astrometry.matcher: Purged 2206 unusable sources, leaving 225 usable sources processCcd.calibrate.astrometry.matcher: Matched 4 sources processCcd.calibrate.astrometry.matcher WARNING: Number of matches is smaller than request processCcd FATAL: Failed on dataId={'taiObs': '2015-01-21', 'pointing': 1116, 'visit': 19696, 'dateObs': '2015-01-21', 'filter': 'HSC-I', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 103, 'expTime': 300.0}: File ""src/sip/CreateWcsWithSip.cc"", line 142, in lsst::meas::astrom::sip::CreateWcsWithSip<MatchT>::CreateWcsWithSip(const std::vector<_RealType>&, const lsst::afw::image::Wcs&, int, const lsst::afw::geom::Box2I&, int) [with MatchT = lsst::afw::table::Match<lsst::afw::table::SimpleRecord, lsst::afw::table::SourceRecord>] Number of matches less than requested sip order {0} lsst::pex::exceptions::LengthError: 'Number of matches less than requested sip order' {code} {code:title=HSC fit: visit=19696 ccd=103} 2016-06-10T17:20:11: processCcd.calibrate.astrometry: Found 84 catalog sources 2016-06-10T17:20:11: processCcd.calibrate.astrometry: Matching to 137/162 good input sources 2016-06-10T17:20:11: processCcd.calibrate.astrometry: Matched 19 sources 2016-06-10T17:20:11: processCcd.calibrate.astrometry WARNING: Number of matches is smaller than request 2016-06-10T17:20:11: processCcd.calibrate.astrometry: 19 astrometric matches for 103, 1_31 2016-06-10T17:20:11: processCcd.calibrate.astrometry: Refitting WCS 2016-06-10T17:20:11: processCcd.calibrate.astrometry: Astrometric scatter: 0.086131 arcsec (with non-linear terms, 18 matches, 1 rejected) {code} Other failed visit/ccd combos: visit=19684 ccd=101 visit=30488 ccd=95: RuntimeError: Unable to match sources This may simply be due to some threshold in the configs that is rejecting more stars on the LSST side, but this is not confirmed. Please investigate the cause of these failures.",6 DM-6533,"LDM-151 adjustments","Adding text to LDM-151 where appropriate, working around the structure defined by Jim et al.",2 DM-6538,"Write DMTN describing Lupton diffim decorrelation","Write a technote describing the analysis and implementation of the Lupton(ZOGY) difference image decorrelation correction. A new technote has been set up, it will be: http://dmtn-021.lsst.io",12 DM-6540,"Propose track to improve container infrastructure","Qserv uses Docker for deployment, this ticket will add track on how to improve container deployment and management.",10 DM-6541,"lsst-dev shared stack should provide release builds","The shared stack on {{lsst-dev}} (etc) currently only provides tagged weekly builds of the LSST stack. Releases, RCs, etc are not included. Please update the build script so that they are. NB simply including these builds is easy enough by changing the {{VERSION_GLOB}} regular expression in {{shared_stack.py}}. However, the {{current}} version is selected by a lexicographic sort of available versions. That works well enough for weekly builds ({{w_2016_XX}} is less than {{w_2016_XX+1}}), but fails with other tags. Better use a sort based on the date the tag was created on the HTTP server instead, perhaps.",1 DM-6542,"Prevent external viewer from popup blockers.","Currently, when external viewer launched, it is blocked by pop-up blockers. Need to change polling logic to a pushed solution so the 'launch' action can happen immediately. ",3 DM-6544,"Release Note integration for 12_0 Stack Release","Transcribe v12_0 release notes prepared by development teams on Confluence into the Pipelines documentation sphinx project. Pipelines documentation is published with LSST the Docs to https://pipelines.lsst.io.",3 DM-6545,"Setup mononode test environment for initial learning about installations","How to load data and perform queries. Investigate DAX interface to qserv.",4 DM-6546,"Add queryId to messages at start and end of user queries.","The queryId, ""QI=xxx:"", needs to be added to log messages that are useful for analysis. of primary interest are messages that indicate that a query has begun or finished, such as ""Discarded UserQuerySelect"".",1 DM-6547,"Capture proposed epic review procedure in DMTN-020","See notes at https://confluence.lsstcorp.org/display/DM/ProjMgmt+Meeting+2016-06-14",1 DM-6548,"Capture release policy in DMTN-020","Capture the policy for releases (all work to be done 2 weeks before end of cycle, release at the cycle changeover) in DMTN-020.",1 DM-6552,"Attend SBAG prep meeting at UW","[~nlust] will travel to UW for preparatory discussions in advance of this month's SBAG meeting.",8 DM-6554,"Take part in LDM-151 Progress Meeting, 2016-06-13",NULL,1 DM-6555,"Take part in LDM-151 Progress Meeting, 2016-06-13",NULL,1 DM-6556,"Take part in LDM-151 Progress Meeting, 2016-06-13",NULL,1 DM-6557,"Take part in LDM-151 Progress Meeting, 2016-06-20",NULL,1 DM-6558,"Take part in LDM-151 Progress Meeting, 2016-06-20",NULL,1 DM-6559,"Take part in LDM-151 Progress Meeting, 2016-06-20",NULL,1 DM-6561,"Fix order of flags in Kron photometry","The flags are not added to the flag handler in the correct order for Kron photometry.",1 DM-6563,"Clean-up rerun documentation","Following DM-4443, there are a few ambiguities in the new {{--rerun}} documentation. Fix them.",1 DM-6566,"Make updateSourceCoords and updateRefCentroids more visible","Implement RFC-197 to make updateSourceCoords and updateRefCentroids more visible",1 DM-6568,"Further prep for SBAG meeting, attend video telecon with Heidi et al.","Read back ground materials on LSST moving object simulations. This will be used to prepare for both the SBAG meeting and to come up with questions that need clarification in the preparatory telecons.",5 DM-6569,"Remove the extra init method from the SourceDetectionTask","SourceDetectionTask defines both {{init(self, schema=None, **kwds)}} and {{\_\_init\_\_(self, schema=None, **kwds)}}. The first exists purely because of a Doxygen bug that makes {{\copydoc \_\_init\_\_}} fail. However, {code} copydoc \_\_init\_\_ {code} works. Remove the non-dunder init method and update the documentation with {code} \copydoc \_\_init\_\_ {code}.",1 DM-6575,"Refactor Known Issues and Metrics pages in Pipelines Docs","Make Known Issues and Metric Report both top-level pages. Link to installation issues from installation page. See https://pipelines.lsst.io/v/DM-6575/index.html",1 DM-6577,"Convert jointcalTask unittest into a validation measure","Now that jointcal has some basic unittests that check whether the relative and absolute astrometry are less than some value, we should convert those tests into validation measures a la [validate_drp|https://github.com/lsst/validate_drp]. This would help us track whether we are actually improving things as we tweak the algorithm and the mappings that we fit.",4 DM-6578,"Initial tests running HTCondor jobs utilizing Shifter","We start with initial tests of Shifter, with the first goal to submit PBS jobs on the Blue Waters test system utilizing Shifter that start HTCondor master/startd daemons on compute nodes. These daemons will communicate to a remote HTCondor central manager (e.g., running on the Nebula OpenStack) and glide-in to join a working pool. The setup will then be tested with simple payload jobs (these submitted from a Nebula instance running the schedd) that verify access to the LSST stack within the UDI (User Defined Image).",4 DM-6580,"Understand and ensure variance plane compliance with diffim decorrelation","Understand how the variance plane should be adjusted in the decorrelation (ZOGY) correction, and ensure it is being done correctly.",8 DM-6581,"Decrease warning messages in dipoleFitTask","The dipoleFitTask was spitting out too many warnings. Change many of those to debug statements and remove the `lmfit` UserWarnings.",1 DM-6582,"Design a metadata system for LSST code and documentation repositories (technote)","This ticket involves the research and design of a metadata system for describing LSST code and documentation repositories. Such metadata would be leveraged by DocHub and LSST the Docs (see [SQR-011|https://sqr-011.lsst.io]) and would reside as a YAML/JSON file in a resource’s GitHub repository. [JSON-LD|http://json-ld.org] is of particular interest. I’m also consulting with GitHub, ADS, Zenodo, and CfA Library on making a sustainable system. *Note: this story should be moved to a DocHub epic.*",3 DM-6588,"Adapt qa analysis script for LSST vs. HSC coadd processing comparison","The analysis script was adapted for single visit processing comparisons in DM-4393 and DM-4730. Do the same here for coadd processing comparisons.",4 DM-6589,"Fill out Software Primitives section of LDM-151",NULL,2 DM-6591,"Implement exception translators in upstream pybind11","Pybind11 does not currently support translation of custom exceptions. This ticket tracks work done on upstream pybind11 (internal fork https://github.com/lsst-dm/pybind11-1) to implement this functionality. It should support functionality equivalent to (but not necessarily with the same API) as Boost Python exception translators (http://www.boost.org/doc/libs/1_61_0/libs/python/doc/html/reference/high_level_components/boost_python_exception_translato.html).",4 DM-6593,"firefly api related issues due to irsa integration.","* firefly_loader.js mistakenly uses relative path to load dependencies when it should resolve it via location of the loading script. * TablePanel should render html content as html by default. * paging bar style does not show correctly in irsa html * row height does not resize to the icon size, the old api did. and the default row selectable is set to false in the old api. * the help button needs to be added on top of the table panel. * the expand button does not function as expected (open a full table panel).",6 DM-6596,"Write command-line driver tutorial for LSST@Europe2 meeting","This will be done as DMTN-023 so the results are preserved for posterity. This may be somewhat redundant with the work Mandeep Gill is doing in translating HSC docs, but I need it now; we can merge later.",1 DM-6598,"Prepare presentation for SPIE","Write the presentation for the SPIE conference. Date of presentation: 26th June.",2 DM-6600,"Clean up naming of multiband tasks and scripts","Several of the multiband processing tasks and files in pipe_tasks and pipe_drivers have inconsistent names: - Some task names do not agree with the script names. - Words like ""Coadd"" and ""Merged"" are not consistently used. Actually making these changes is trivial, but the work also requires creating and shepherding an RFC.",1 DM-6601,"Port change to EXP-ID handling","From [HSC-1409|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1409]: {quote} Due to an operational reason (to meet the requirement of Subaru FITS dictionary), the definition of EXP-ID is soon to be changed in the data acquisition side. In the new definition, EXP-ID is set to 'HSCE%08d' where the letter 'E' is fixed as requested in the dictionary, and the number part corresponds to exactly the same number as our familiar 'visit'. obs_subaru:ingest.py needs to be updated to include this rule. The data taken with this change so far are: HSCA07441200--HSCA07441757 HSCA90925200--HSCA90929557 {quote} The change made as part of HSC-1409 introduces a new code path for the updated data, while old data continue to be supported with the old code path.",1 DM-6603,"Reporting improvements","read and critique Jacek's LPM document and the more manual oriented work from John Swinbank. Phone con with Kevin w.r.t. reporting channel for equipment expenses in Jira (as opposed to the now clear separate distinct financial channel). Worked out checklist and principals for revised WBS. ",6 DM-6607,"Install packstack to test OpenStack Object Storage API",NULL,2 DM-6608,"Finalize v12 Pipelines release documentation","Add the release announcement and finalize other documentation details in pipelines.lsst.io for the v12 release.",1 DM-6609,"Review LDM-135 (LSST Database Design)",NULL,4 DM-6610,"Further refine alert generation pipelines sections","There is much more information in the document, but the pipelines sections need to be refined. We also need to give the software primitives a go over.",20 DM-6611,"Update X16/W16 release notes for qserv and dax services",NULL,2 DM-6612,"Make HSC processing without bright object catalogs easier","obs_subaru enables bright object masks by default, as that's desirable for HSC production runs. However, when HSC data is processed without bright object masks available (as will happen in most GO observations and development use), multiBandDriver.py will fail because the BRIGHT_OBJECT mask plane is not present but the base_PixelFlags algorithm is configured to make use of it. This is confusing, and it also requires the definition of a configuration file to fix the problem because base_PixelFlags cannot be configured directly on the command-line. Some possibilities for fixing this: - Add the BRIGHT_OBJECT mask plane in AssembleCoadd if doMaskBrightObjects is True but the external catalog is not found. This will make the PixelFlags operation a silent no-op. - Allow configuration options to allow PixelFlags algorithm to silently skip some flags if the appropriate masks are not available. I am sure there are other options as well. ",2 DM-6614,"Include Kron parameters in algorithm metadata","The Kron code doesn't set the algorithm metadata. E.g. {code} algMetadata.set(""ext_photometryKron_KronFlux_nRadiusForFlux"", config.plugins[""ext_photometryKron_KronFlux""].nRadiusForFlux) {code} ",1 DM-6616,"update ""newinstall.sh"" nebula images & docker containers - v12_0",NULL,1 DM-6620,"Cannot instantiate LoadAstrometryNetObjectsTask without Config object","One should be able to create a LoadAstrometryNetObjectsTask without passing a Config object, if one only wants the default configuration. Currently it raises TypeError: {code} Traceback (most recent call last): File ""testJointcal.py"", line 79, in setUp refLoader = LoadAstrometryNetObjectsTask() TypeError: __init__() takes at least 2 arguments (1 given) {code} If the config object really is a kwarg, it should default None and create a default config, so that one doesn't have to do, e.g.: {code} LoadAstrometryNetObjectsTask(LoadAstrometryNetObjectsConfig()) {code}",1 DM-6621,"cleanup non-survey-generic python in jointcal","jointcal.py current does things like: {code} for dataRef in dataRefs: if dataRef.dataId[""visit""] == int(visit) and dataRef.dataId[""ccd""] == int(ccd): ... {code} This is not survey generic, and is probably not the best way to identify data blocks anyway. This, and other non-generic things in jointcal.py should be cleaned up so they work across surveys.",4 DM-6622,"make jointcal integration/validation test for hsc","We need an integration/validation test for jointcal on hsc data, to show that jointcal can run safely on hsc data processed through the stack.",10 DM-6623,"make jointcal integration/validation test for cfht","We need an integration/validation test for jointcal on cfht data, to show that jointcal can run safely on cfht data processed through the stack.",10 DM-6624,"make jointcal integration/validation test for DECam","We need an integration/validation test for jointcal on DECam data, to show that jointcal can run safely on DECam data processed through the stack.",10 DM-6625,"make jointcal integration/validation test for lsstSim","We need an integration/validation test for jointcal on lsstSim data, to show that jointcal can run safely on lsstSim data processed through the stack.",10 DM-6627,"Fix base_* stuff in CcdImage.cc","CcdImage.cc currently has hard-coded a bunch of {{getSchema().find(""base_blah"").key}} things. These should either be replaced with ""slot_*"", config.blahName, or dealt with at a higher level (e.g. not loading all those values directly inside of ccdImage::LoadCatalog). Once this is done, we should delete the comments at the top of the file.",1 DM-6628,"Reimplement diffim decorrelation as task","Reimplement the image decorrelation as a subtask rather than a direct call to a function.",6 DM-6629,"validate_drp: design and implement an API for metric measurements and serializations","{{validate_drp}} computes metrics and generates JSON that, through the [post-qa|https://github.com/lsst/post-qa] tool, is submitted to the SQuaSH REST API for persistence and display in a web app. A previous ticket, DM-6086, we bolted on a JSON serialization scheme compatible with SQuaSH. However, this approach was not well integrated with {{validate_drp}}. We want a framework/API where serialization is handled consistently and integrally with metric computations. This includes the semantic serialization of computational parameters and reduced datasets. This API can be applied beyond {{validate_drp}} as a means for metrics and integration tests to be submitted to SQuaSH as well.",22 DM-6630,"Support ingesting reference catalogs from FITS files","Support a means of ingesting index reference catalogs from FITS tables (e.g. SDSS catalogs).",2 DM-6631,"Single-frame processing tasks are no longer usable without a Butler","Adding a butler argument to the constructor signatures for {{CharacterizeImageTask}}, {{CalibrateTask}}, and {{ProcessCcdTask}} makes these tasks difficult to use without a butler. The fix is to make the butler argument optional (with a default of None), while adding another argument that allows a fully-constructed reference object loader to be provided directly instead. This is closely related to DM-6597, which has the opposite problem: pipe_drivers' {{SingleFrameDriverTask}} doesn't take a butler argument, but it needs to in order to provide one to {{ProcessCcdTask}}. I have a fix for this just about ready, but I'd like to add some unit tests that verify we can run all of these tasks both from the command-line and directly before calling it complete.",3 DM-6632,"Make match and flag propagation more reusable","We have two bits of code for doing spatial matches and propagating flags: - {{PropagateVistFlagsTask}}: propagates flags from individual visit catalogs to coadd catalogs, and depends on a butler to do so (reasonably; it includes the smarts to load the appropriate catalogs, so it has to do I/O). - {{CalibrateTask.copyIcSourceFields}}: propagates fields from icSrc to src, but is only usable as part of {{CalibrateTask}}. Both of these should delegate at least some of their work to new class (possibly a Task) that manages the Schemas, SchemaMappers, and cross-matching necessary to do this work. This new class should be reusable without a butler and without constructing any higher-level tasks.",4 DM-6633,"HSC ISR configuration file is applied to ProcessCcdTask, not IsrTask","{{obs_subaru/config/hsc/isr.py}} has its config options specified relative to {{ProcessCcdTask}}'s config hierarchy, not {{IsrTask}}'s. This allows the ISR task to be retargeted in this file, but it will prevent {{IsrTask}} from being run as a {{CmdLineTask}} directly. ISR Task retargeting should be moved to {{config/processCcd.py}}, allowing the {{config/isr.py}} level to be moved to the appropriate level.",1 DM-6634,"Add JIRA-wrangling howto to DMTN-020","Expand https://dmtn-020.lsst.io/v/DM-6447/#jira-maintenance to describe best practices for T/CAMs working with JIRA. Include: * Appropriate labels; * Teams; * ... other things?",1 DM-6638,"LTD Keeper: Auto slug for edition paths deals with underscores","Had a bug where {{utils.auto_slugify_edition}} did not replace underscores with a dash, and therefore failed {{utils.validate_path_slug}}. This created a silent breaked where a branch like {{u/rowen/r12_patch1}} did not get an edition created for it. This ticket adds this replacement code and adds a test for such a case.",1 DM-6640,"IsrTask is not a valid CmdLineTask","IsrTask is a command-line task, but its run method does not take a dataRef (it instead has a {{runDataRef}} method. This is inconsistent with other {{CmdLineTasks}} and more importantly breaks {{parseAndRun}}. I'm committing a small workaround on DM-6631 to get {{parseAndRun}} working, but the ultimately method names should be made consistent across CmdLineTasks. That will require an API change and hence an RFC.",1 DM-6642,"Make list of elements for consideration for planning packages","Create a detailed checklist for developing the planning packages for the replan and WBS restructuring.",2 DM-6643,RADICAL-Pilot,"Review [RADICAL-Pilot|http://radicalpilot.readthedocs.io/en/latest/index.html] against criteria defined in the epic.",6 DM-6644,Makeflow,"Review Makeflow against criteria defined in the epic. http://ccl.cse.nd.edu/software/makeflow/",6 DM-6645,pinball,"Review [pinball|https://github.com/pinterest/pinball] workflow management system.",6 DM-6646,CloudSlang,"Review CloudSlang against criteria defined in the epic. http://cloudslang-docs.readthedocs.io/en/v0.9.60/index.html",6 DM-6647,"Adapt qa analysis script to apply corrections measured by meas_mosaic","DM-2674 involves getting HSC's {{meas_mosaic}} working with the LSST stack. This issue consists of adapting the analysis.py script of DM-4393 & DM-4730 to (optionally) apply the astrometric and photometric solutions derived running {{meas_mosaic}} to the individual visits before comparison. This is useful in general and is specifically useful in comparing the {{meas_mosaic}} results between the HSC and LSST stacks.",2 DM-6650,"Management level review of two products of the Management process working group","Reviewed https://github.com/lsst/LDM-PMT/blob/integration/index.rst and https://dmtn-020.lsst.io/v/DM-6447/ Made extensive markup of LDM-PMT, delivered to Mario Juric. Assessed DM-6447, which show promise of an actual workable manual, though not complete.",1 DM-6651,"Move new reference loader so meas_astrom can use it and perform some cleanup","The new reference object loader code lives in pipe_tasks, which means it cannot be directly used by code in meas_astrom. This will hamper separating astrometry.net out of meas_astrom, because unit tests need reference catalogs and meas_astrom cannot depend on pipe_tasks. Also, I'd like to take a cleanup pass on the module names, so the new code is easier to find, and improve the unit tests.",2 DM-6652,"Remove database hack","DM-5988 introduced a hack in reading the raw files: we use a database to cache metadata from the shutter files and update the camera files at read time. The camera files have now been ""sanitised"" (updated with the appropriate metadata), and it's time to remove the hack. [~mfisherlevine] writes: {quote} Data is on lsst-dev in: /nfs/lsst2/photocalData/data/monocam/sanitised9/1m3/1m3 Raw calibs are in: /nfs/lsst2/photocalData/data/monocam/sanitised9/1m3/calibs Regarding what I want: everything to be the same, but with a normal ingest, i.e. no splicing, just taking everything that is needed from one set of files. Some points to note: * should be able to ingest all the raws and calibs files, and register their OBJECT types to allow processing with these as ids (inc. pipe_drivers scripts) * pipe_drivers master calib scripts should still run (and their outputs still be ingestable) * processCcd should run {quote}",2 DM-6653,"implement the active target","When a dialog such as catalog search is displayed, it should be able to pick up the active target or the coordinates from a highlighted row in a table. Please, implement the mechanism that will automatically pick up those coordinates and pre-fill the search form for you.",6 DM-6656,"ffApi image related issues found by irsa integration","*-for external image viewer, the default RangeValues causes problem, i.e. other defaults not set-. (FIXED) *-global default does not always apply to external image viewer-(FIXED DM-7016) The Gator implementation related to coverage map (1) default symbol size, shape, color setting is different from that of original map (2) cannot specify the shape, size, and color through API; (3) cannot specify the shape, size, and color of a search center through API; (4) -does not display any image and source when the table has only one ra,dec values, for example: one table with one position value or one table with many records but has the same ra,dec values.- MOVED to [DM-7001] (5) -the sources on coverage map are not clickable. However, on table and plot are clickable and work fine.- (FIXED) ",6 DM-6657,"ffApi XYplot related issues found by irsa integration","* default symbol size, shape,and color setting is different from that of original version. * no XY Plot Options pop-out windows * the plot displays non-ascii characters on the panel (for example:  Fit  ) * miss Filter Dialog on the plot panel comparing with the original version. * does not accept default column names for the plot.",6 DM-6660,"CR finder does not care about XY0 of input image","Port of [HSC-1391|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1391]: {quote} The current version of CR finder does not care about XY0 of the input image and when I try to run CR finder on warped (difference) image, PSF cannot be properly extracted. {quote} and: {quote} I have noticed that the center of warped image is a gap between CCDs and PSF estimation there will fail. So get PSF without specifying the position is good enough. PSF class will select the best position. {quote}",1 DM-6661,"ConfigDictField says ""Inequality in keys for..."" even if I give 2 same configurations","From [HSC-1401|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1401]: {quote} config.py: {code:python} from lsst.meas.photocal.colorterms import ColortermGroupConfig for key in ['i', 'i2', 'y', 'r', 'N1', 'N2', 'N3', 'z']: root.calibrate.photocal.colorterms.library[key] = ColortermGroupConfig.fromValues({}){code} This comamnd line {code:bash} rm -fr output ; for i in {1..2} ; do processCcd.py ./HSC --output output -C config.py ; done {code} raises following error {noformat} 2016-06-01T02:43:45: processCcd FATAL: Comparing configuration: Inequality in keys for calibrate.photocal.colorterms.library: ['z', 'i', 'i2', 'r', 'y', 'N1', 'N2', 'N3'] != ['N3', 'i', 'i2', 'r', 'y', 'N1', 'N2', 'z'] 2016-06-01T02:43:45: processCcd FATAL: Failed in task initialization: Config does match existing config on disk for this task; tasks configurations must be consistent within the same output repo (override with --clobber-config) {noformat} {quote}",1 DM-6663,"Study iPlant as a potential candidate for workspace implementation",NULL,2 DM-6664,"Investigate why afw.table.IdFactory doesn't allow reserved=0","Setting reserved=0 when constructing a source ID factory (as would be logical when there is no exposure ID to reserve bits for) strangely doesn't work; it seems to be necessary to reserve at least one bit. This may be a signedness problem (we use signed 64-bit integers for IDs to appease FITS, which is unfortunate), but we should be careful just reducing the number of available bits, as this could break code that expect to read IDs already written to disk. Note that any change to this code in afw.table may require changes to code in daf.butlerUtils.ExposureIdInfo as well.",2 DM-6665,"set up unit test for projection in Java","While working on DM-6438 (set up unit test for projection in JavaScript), we realized we should have a parallel unit test system set up for Java code, to keep the two systems in sync. ",6 DM-6667,"Data Backbone conops iteration 4: submit to TCT","Submit the document for TCT change control. Process is TBD. If it is not accepted by TCT, further work is not in the scope of this epic, and would need to be planned in the EV system.",1 DM-6668,"Data Backbone conops: develop engineering considerations for BOE for work package","Based on the data backbone services conops, develop a list of engineering considerations for making a BOE for the data backbone planning package.",3 DM-6669,"Authentication & Authorization conops iteration 1: create raw draft (internal)","Write a raw draft of the concept of operations for authentication and authorization services. In this iteration the document is developed in Google docs following the ConOps template.",5 DM-6670,"Authentication & Authorization conops iteration 2: group review to produce first draft","Review raw draft of concept of operations for the AA services to work through underdeveloped areas, clear up uncertainties, and make readable.",2 DM-6671,"Authentication & Authorization conops iteration 3: larger review to produce second draft","Review first draft of AA services conops within Data Processing Architecture working group, bringing in relevant experts. Input from review is incorporated into a second draft. ",6 DM-6672,"Authentication & Authorization conops formatting: convert second draft to reStructuredText","When the AA services conops is in a solid state, convert the Google doc to reStructuredText following DM's documentation versioning process.",2 DM-6673,"Authentication & Authorization conops iteration 4: submit to Systems Engineering","Submit the document for TCT change control. Process is TBD. If it is not accepted by TCT, further work is not in the scope of this epic, and would need to be planned in the EV system.",1 DM-6674,"Authentication & Authorization conops: develop engineering considerations for BOE for work package","Based on the AA services conops, develop a list of engineering considerations for making a BOE for the AA planning package. ",3 DM-6675,"Level 3 Hosting conops iteration 1: create raw draft (internal)","Write a raw draft of the concept of operations for Level 3 Hosting services. In this iteration the document is developed in Google docs following the ConOps template.",4 DM-6676,"Level 3 Hosting conops iteration 2: group review to produce first draft","Review raw draft of concept of operations for the L3 Hosting services to work through underdeveloped areas, clear up uncertainties, and make readable.",2 DM-6677,"Level 3 Hosting conops iteration 3: larger review to produce second draft","Review first draft of Level 3 Hosting services conops within Data Processing Architecture working group, bringing in relevant experts. Input from review is incorporated into a second draft. ",6 DM-6678,"Level 3 Hosting conops formatting: convert second draft to reStructuredText","When the L3 Hosting services conops is in a solid state, convert the Google doc to reStructuredText following DM's documentation versioning process.",2 DM-6679,"Level 3 Hosting conops iteration 4: submit to TCT","Submit the document for TCT change control. Process is TBD. If it is not accepted by TCT, further work is not in the scope of this epic, and would need to be planned in the EV system.",1 DM-6680,"Level 3 Hosting conops: develop engineering considerations for BOE for work package","Based on the L3 Hosting services conops, develop a list of engineering considerations for making a BOE for the L3 Hosting planning package.",3 DM-6681,"Batch Processing for commissioning conops iteration 1: create raw draft (internal)","Write a raw draft of the concept of operations for batch processing services for the commissioning phase. In this iteration the document is developed in Google docs following the ConOps template.",4 DM-6682,"Batch Processing for commissioning conops iteration 2: group review to produce first draft","Review raw draft of concept of operations for the batch processing for commissioning services to work through underdeveloped areas, clear up uncertainties, and make readable.",2 DM-6683,"Batch Processing for commissioning conops formatting: convert first draft to reStructuredText in a technical note","When the batch production for commissioning services conops is in a solid state, convert the Google doc to a DM technical note in reStructuredText.",1 DM-6684,"Batch Processing for commissioning conops: develop engineering considerations for BOE for work package","Based on the batch services for commissioning services conops, develop a list of engineering considerations for making a BOE for the batch services planning package.",3 DM-6685,"Planning package for Management, Engineering and Integration with engineering judgement BOE based on RACI diagram","Following list of elements for consideration (DM-6642), estimate planning packages for Management, Engineering and Integration WBS element. BOE is derived from RACI document, which list roles and responsibilities of line management, reporting group, steering group, and area technical leads.",1 DM-6686,"Planning package for L1 Services with engineering judgement BOE","Following list of elements for consideration (DM-6642), estimate planning packages for Level 1 Services WBS element. BOE is derived from detailed plan for prompt processing and archiving services created in February and engineering judgement based on conops documents.",1 DM-6687,"Planning package for Batch Production Services with engineering judgement BOE","Following list of elements for consideration (DM-6642), estimate planning packages for Batch Production Services WBS element. BOE is derived from engineering judgement based on conops documents. ",1 DM-6688,"Planning package for Data Backbone Services with engineering judgement BOE","Following list of elements for consideration (DM-6642), estimate planning packages for Data Backbone Services WBS element. BOE is derived from engineering judgement based on conops documents. ",1 DM-6689,"Planning package for Data Access Hosting Services with engineering judgement BOE","Following list of elements for consideration (DM-6642), estimate planning packages for Data Access Hosting Services WBS element. BOE is derived from engineering judgement based on conops documents.",1 DM-6690,"Planning package for Common Workflow/Middleware with engineering judgement BOE","Following list of elements for consideration (DM-6642), estimate planning packages for Common Workflow/Middleware WBS element. BOE is derived from engineering judgement based on conops documents.",1 DM-6691,"Planning package for Misc. Services with engineering judgement BOE","Following list of elements for consideration (DM-6642), estimate planning packages for Miscellaneous Services WBS element. An example is the Authentication and Authorization services. BOE is derived from engineering judgement based on conops documents.",1 DM-6692,"Planning package for Development Support Services with engineering judgement BOE","Following list of elements for consideration (DM-6642), estimate planning packages for Development Support Services WBS element. BOE is derived from engineering judgement.",1 DM-6693,"Planning package for ITC and Fabric Provisioning and Operation with engineering judgement BOE","Following list of elements for consideration (DM-6642), estimate planning packages for ITC Fabric Provisioning and Operation WBS element. BOE is derived from engineering judgement. ",1 DM-6694,"Planning package for Service Management with engineering judgement BOE","Following list of elements for consideration (DM-6642), estimate planning packages for Service Management WBS element. BOE is derived from engineering judgement based on ITIL methodology.",1 DM-6695,"Submit change request","Submit formal change request to restructure NCSA WBS in PMCS.",1 DM-6696,"Revise Level 1 ConOps ","Revise the Level 1 conops to incorporate the minimal required functionality of Level 1 Services: minimal data archiving of camera data and minimal transport via data backbone to NCSA.",4 DM-6697,"Build draft design","Produce functional design breakdown from the revised conops (DM-6696).",16 DM-6698,"Articulate the design in format needed for planning","Articulate the design created in (DM-6697) into the format needed for planning.",12 DM-6699,"Produce revised WBS","Based on articulated design, revise WBS to incorporate phase.",10 DM-6700,"Discuss elements of RFC","Discuss elements of RFC (technical details and scope).",6 DM-6701,"Produce RFC","Write up and submit RFC.",2 DM-6702,"Respond to RFC comments and update RFC as needed","Respond to RFC comments and update RFC as needed.",2 DM-6705,"Select workflow based on conops and review of workflow systems","Based on use cases/requirements gathered in DM-6270 and evaluation reports completed in DM-6276, select workflow system.",2 DM-6706,"Discuss elements of RFC","Discuss elements of workflow RFC (technical details, scope, requirements).",4 DM-6707,"Produce RFC","Write up and submit RFC.",2 DM-6708,"Respond to RFC comments and update RFC as needed","Respond to RFC comments and update RFC as needed.",2 DM-6709,"Pull down and install OCS SAL code in prep for ConOps development","At the Camera Workshop in Mid-June, OCS Team members suggested that DM pull down their Service Abstraction Layer software and gain familiarity with it. The User manual is being studied before compiling the software and running it with DM software as a means of simulating planned Telescope & Site processes and how DM will interact with them. Most of the work for this epic will be conducted in August. This story captures our prep work.",4 DM-6710,"Monitoring plan for Startup procedure","Identify startup processes to be monitored for health and to provide notification for startup failure.",1 DM-6711,"L1 entity prototypes","This story addresses the need to separate the processes that connect to the DAQ and retrieve the image data from the processes that forward the image data to NCSA. Requirements for this component and the component that formats the image data into a file which includes associated metadata were discussed at length during the Camera Workshop this month. Prototypes for these component processes are underway.",16 DM-6712,"Message Dictionary additions","Message types for system bookkeeping acknowledgements as well as report messages were added to the existing dictionary and the means for acting upon these message types are being added to component prototype code. In addition, needed changes were made to the existing dictionary so all reporting entities write more complete details to their report message queues.",4 DM-6713,"Amendments to message interaction","Proper acknowledgements began being added to the messaging system this month.",6 DM-6714,"Camera Workshop attendance","Work on preliminary specific additions to the camera interaction ConOps took place this month during attendance at the Camera Workshop",6 DM-6715,"Use Shifter+HTCondor in processing Stripe82 ref data at modest scale","To test out processing at modest scales (~ 100 -- 1000 cores) utilizing Shifter+HTCondor on machines like BW, organize processing (processCcd of obs_sdss) of stripe82 data similar to that used in the lsst_dm_stack_demo (run=4192 field=300).",6 DM-6723,"Add tests for order of flags to all measurment plugins","In the meas_base framework, we independently define an enumeration of available flags [(e.g.)|https://github.com/lsst/meas_extensions_photometryKron/blob/cba01575dab0cd609c7e2a3f3d08632b94f97f58/include/lsst/meas/extensions/photometryKron.h#L82] and a set of table fields for storing flags [(e.g.)|https://github.com/lsst/meas_extensions_photometryKron/blob/cba01575dab0cd609c7e2a3f3d08632b94f97f58/src/KronPhotometry.cc#L422]. We implicitly assume that these are declared in the same order, but do not, in general, enforce this. In DM-6561, these were found *not* to be in the same order in meas_extensions_photometryKron. Setting a flag based on a bad result would therefore set the wrong flag in the output table. In the DM-6561 solution, we introduced a test for this which is specific to the photometryKron codebase. However, the basic structure of the test would be easily extended to cover all meas_base plugins to ensure this error can never occur. Do so.",3 DM-6725,"Message refinement , in light of development #1",NULL,12 DM-6726,"Default chart and other optimizations","These are the changes to support defalt chart and single chart type (as for IRSA release) - Remove chart selection from chart area - Use dropdown for chart selection (can be omitted if single chart type is used) - Populate current values in chart options - Support Clear and Reset in chart options - For tables, connected to charts, if no default parameters are specified, default chart is an XY plot with CATALOG_COORD_COLS (used to produce an overlay) for catalogs or two first numeric columns for other tables. - Label, matching column expression, and unit, matching table model, are default parameters for both app and api now. ",6 DM-6727,"Message refinement #2",NULL,12 DM-6728,"Camera workshop attendance","Attend camera workshop, Meeting did not fully address the need. Travel to SLAC.",8 DM-6729,"Summarize meeting results into Concepts of Operation",NULL,2 DM-6730,"ConOps for Comfort Console and System Monitor","This is the first definition of the Concept of Operation of the Comfort Console and System Monitor piece of the DM system. This application will play a key role in fault detection and correction as well as monitor actively the state (and sub-state) of all the components in the DM system. Based on the role of the operator, he / she will be able to dig down into the faults and take corrective action. An action dashboard will provide a hierarchical view of the state of the system and its components at any point in time.",20 DM-6731,"Definitions of Alarms and Actions","This is a task to define all the failure cases and alarms that they should generate. It will also define the action that will be taken in the event of an alarm / fault and who will be taking the action.",30 DM-6732,"Process reference data on ""lsstdev pool"" for reference",NULL,2 DM-6733,"Reference processing on NERSC Cori Shifter implementation",NULL,4 DM-6736,"Write test jobs and submit files",NULL,2 DM-6737,"Investigate HTCondor configuration wrt dropping of nodes in backfill scenario",NULL,4 DM-6738,"Run test jobs and evaluate",NULL,2 DM-6740,"Write final report",NULL,2 DM-6741,"Make SuperTask data-aware",NULL,10 DM-6742,"Add workflow features",NULL,10 DM-6743,"Select existing tasks for prototyping conversion to workflow supertask",NULL,1 DM-6744,"Convert selected tasks","Convert a {{CmdLineTask}} into {{SuperTask}}",10 DM-6745,"Finish gathering input from DM representatives",NULL,3 DM-6746,"Compile input",NULL,2 DM-6747,"Review conops template",NULL,1 DM-6748,"Iteration 1: Write raw draft based on input gathered from DM representatives",NULL,6 DM-6749,"Iteration 2: group review to produce first draft",NULL,6 DM-6750,"Iteration 3: larger review to produce second draft",NULL,8 DM-6751,"Iteration 4: final draft and convert to reStructuredText to produce tech note",NULL,4 DM-6752,"Service Management for F16 June","Dividing F16 Service Management ~ monthly.",4 DM-6753,"Service Management for F16 July"," Dividing F16 Service Management ~ monthly. ",4 DM-6754,"Service Management for F16 August"," Dividing F16 Service Management ~ monthly.",4 DM-6755,"Write test programs to exercise Swift API with OpenStack ",NULL,2 DM-6756,"Write test programs to exercise Swift API with Ceph",NULL,2 DM-6757,"Benchmark Swift command line tool for objects less than 5GB",NULL,4 DM-6758,"Benchmark Swift command line tool for objects greater than 5GB",NULL,4 DM-6759,"Benchmark Swift custom tool for objects less than 5GB",NULL,4 DM-6760,"Benchmark Swift custom tool for objects greater than 5GB",NULL,4 DM-6761,"Analyze results and write report",NULL,4 DM-6762,"Find and read documentation for OpenStack Swift API",NULL,1 DM-6763,"Find and read documentation for Ceph API",NULL,1 DM-6764,"Write abstract API",NULL,4 DM-6765,"Research existing API",NULL,2 DM-6766,"Find and read documentation for DDN WOS API",NULL,1 DM-6768,"Create description of features in storage APIs",NULL,2 DM-6769,"Write test programs to exercise Swift API with DDN WOS",NULL,2 DM-6770,"Benchmark Swift command line tool for objects less than 5GB",NULL,2 DM-6771,"Benchmark Swift command line tool for objects greater than 5GB",NULL,2 DM-6772,"Benchmark Swift custom tool for objects less than 5GB",NULL,2 DM-6773,"Benchmark Swift custom tool for objects greater than 5GB",NULL,2 DM-6774,"Analyze results and write report ",NULL,3 DM-6775,"Write test programs to exercise object stores",NULL,6 DM-6776,"Benchmark Swift command line tool for objects less than 5GB",NULL,2 DM-6777,"Benchmark Swift command line tool for objects greater than 5GB ",NULL,2 DM-6778,"Benchmark Swift custom tool for objects less than 5GB",NULL,2 DM-6779,"Benchmark Swift custom tool for objects greater than 5GB ",NULL,2 DM-6780,"Write report ",NULL,6 DM-6781,"remove SizeMagnitudeStarSelector","The sizeMagnitudeStarSelector is still in meas_algorithms, but it is unused and likely no longer works. We should either remove it, or update it to be fully supported. The same holds true for any other C++-based star selectors we still have lying around.",1 DM-6783,"Add support for deriving from Python exception types to pybind11","DM-6302 adds support for custom exception translators to pybind11. However exceptions mapped do not inherit from Python {{BaseException}} or higher. This prevents exceptions from being raised and caught with {{except Exception as e}} in Python. This behaviour also occurs with Boost Python and Swig (we hack around it with a pure Python wrapper). This ticket aims to solve the problem by adding support for inheritance from Python exception types to pybind11.",4 DM-6784,"Port meas_extensions_convolved from HSC","HSC has a new measurement extension: meas_extensions_convolved. This performs aperture photometry with the PSF degraded to nominated seeings (similar to how galaxy photometry is commonly done these days). Relevant HSC tickets are [HSC-1395|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1395] and [HSC-1408|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1408].",5 DM-6785,"Port parent/child measurement from HSC","The deblender sometimes gets into trouble with cluster galaxies, and the deblended fluxes aren't accurate. In that case it helps to have measurements on the image without any deblending having been performed. This is a feature used in HSC's mid-2016 production run afterburner, ticket [HSC-1400|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1400]. This feature should be ported for use in LSST.",5 DM-6788,"Document meas_extensions_ngmix","meas_extensions_ngmix has no useful documentation, not even a {{doc}} directory. Add some. This should include at least an overview of the package contents, a description of its capabilities, and instructions on enabling it within the meas_base framework. The package should have a README.",2 DM-6789,Provisioning,NULL,15 DM-6790,"Product Acceptance",NULL,15 DM-6791,"Disaster Recovery Implementation",NULL,20 DM-6792,Documentation,NULL,5 DM-6793,"Capability Validation",NULL,5 DM-6794,"Security Vetting",NULL,5 DM-6795,"Acceptance by Stakeholders",NULL,10 DM-6796,"Capability Design",NULL,10 DM-6797,"Gathering product pricing",NULL,10 DM-6798,"Updating LDM-143",NULL,5 DM-6799,"Acceptance into baseline",NULL,5 DM-6800,Design,NULL,10 DM-6801,Implementation,NULL,10 DM-6802,"Capability Design",NULL,15 DM-6803,Procurement,NULL,10 DM-6804,"Reception and Placement",NULL,10 DM-6805,"Networking Configuration",NULL,10 DM-6806,Provisioning,NULL,25 DM-6807,"Disaster Recovery Implementation",NULL,10 DM-6808,Documentation,NULL,10 DM-6809,"Capability Validation",NULL,20 DM-6810,"Security Vetting",NULL,10 DM-6811,"Acceptance by Stakeholders",NULL,10 DM-6812,"Qserv container crashes on Openstack using up to date CentOS/docker setup"," {code:bash} [qserv@lsst-fabricejammes-qserv-0 ~]$ docker run -it --net=host -e ""QSERV_MASTER=lsst-fabricejammes-qserv-0"" qserv/qserv:dev_master bash qserv@lsst-fabricejammes-qserv-0:/qserv$ /qserv/run/bin/qserv-start.sh INFO: Qserv execution directory : /qserv/run Starting MySQL [FAIL.] Manager of pid-file quit without updating file. ... failed! [FAILing xrootd.[....] : Manager of pid-file quit without updating file. ... failed! failed! See startup logfiles : /qserv/run/var/log/xrootd-console.log, /qserv/run/var/log/worker/xrootd.log [FAILing cmsd.[....] : Manager of pid-file quit without updating file. ... failed! failed! See startup logfiles : /qserv/run/var/log/xrootd-console.log, /qserv/run/var/log/worker/cmsd.log [ ok ing mysql-proxy.. [FAILing qserv-watcher failed! See startup logfile : /qserv/run/var/log/qserv-watcher.log [ ok ing qserv-wmgr. # Here error log can be different sometimes... qserv@lsst-fabricejammes-qserv-0:/qserv$ cat /qserv/run/var/log/mysqld.log ... 2016-06-27 23:50:00 140703880873792 [Note] InnoDB: Waiting for purge to start 2016-06-27 23:50:00 140703880873792 [Note] InnoDB: 5.6.27 started; log sequence number 1661735 2016-06-27 23:50:00 140703880873792 [Note] Plugin 'FEEDBACK' is disabled. 2016-06-27 23:50:00 140703105521408 [Note] InnoDB: Dumping buffer pool(s) not yet started 2016-06-27 23:50:00 140703880873792 [Note] Server socket created on IP: '::'. 2016-06-27 23:50:00 140703880873792 [Note] /qserv/stack/Linux64/mariadb/10.1.11.lsst2/bin/mysqld: ready for connections. Version: '10.1.11-MariaDB' socket: '/qserv/run/var/lib/mysql/mysql.sock' port: 13306 Source distribution 2016-06-27 23:50:04 140703665879808 [Note] /qserv/stack/Linux64/mariadb/10.1.11.lsst2/bin/mysqld: Normal shutdown 2016-06-27 23:50:04 140703665879808 [Note] Event Scheduler: Purging the queue. 0 events 2016-06-27 23:50:04 140703088736000 [Note] InnoDB: FTS optimize thread exiting. 2016-06-27 23:50:04 140703665879808 [Note] InnoDB: Starting shutdown... 2016-06-27 23:50:06 140703665879808 [Note] InnoDB: Shutdown completed; log sequence number 4432991 2016-06-27 23:50:06 140703665879808 [Note] /qserv/stack/Linux64/mariadb/10.1.11.lsst2/bin/mysqld: Shutdown complete 160627 23:50:06 mysqld_safe mysqld from pid file /qserv/run/var/run/mysqld/mysqld.pid ended 160629 19:36:11 mysqld_safe Starting mysqld daemon with databases from /qserv/data/mysql 2016-06-29 19:36:11 139694065747776 [Note] /qserv/stack/Linux64/mariadb/10.1.11.lsst2/bin/mysqld (mysqld 10.1.11-MariaDB) starting as process 143 ... 2016-06-29 19:36:12 139694065747776 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2016-06-29 19:36:12 139694065747776 [Note] InnoDB: The InnoDB memory heap is disabled 2016-06-29 19:36:12 139694065747776 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2016-06-29 19:36:12 139694065747776 [Note] InnoDB: Memory barrier is not used 2016-06-29 19:36:12 139694065747776 [Note] InnoDB: Compressed tables use zlib 1.2.8 2016-06-29 19:36:12 139694065747776 [Note] InnoDB: Using SSE crc32 instructions 2016-06-29 19:36:12 139694065747776 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2016-06-29 19:36:12 139694065747776 [Note] InnoDB: Completed initialization of buffer pool 2016-06-29 19:36:12 139694065747776 [ERROR] InnoDB: ./ibdata1 can't be opened in read-write mode 2016-06-29 19:36:12 139694065747776 [ERROR] InnoDB: The system tablespace must be writable! 2016-06-29 19:36:12 139694065747776 [ERROR] Plugin 'InnoDB' init function returned error. 2016-06-29 19:36:12 139694065747776 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 2016-06-29 19:36:12 139694065747776 [Note] Plugin 'FEEDBACK' is disabled. 2016-06-29 19:36:12 139694065747776 [ERROR] Unknown/unsupported storage engine: InnoDB 2016-06-29 19:36:12 139694065747776 [ERROR] Aborting {code}",10 DM-6813,"Track statistics about user queries and tasks running on chunks",NULL,9 DM-6814,"Move queries to different scheduler if too slow",NULL,8 DM-6815,"Update LSST full-stack processing configuration to match best practice from HSC","In preparation for running an end-to-end comparison of large scale processing with the HSC and LSST stacks, we need to update the configuration to reflect currently understood best practice. In general, we expect the default HSC configuration to be better understood and ""battle-tested"" given that it has been used for science-grade data releases. Audit the default configuration of the full LSST stack (from ProcessCcdTask through multiband coadd processing). Where LSST defaults differ from HSC, update the LSST configuration to match the HSC equivalent unless there's a clear reason why LSST's default should be different. When it's not appropriate to update the LSST configuration, add an override to obs_subaru. In some cases, the LSST and HSC stacks have diverged so that a direct transfer of configuration options isn't possible. Where an equivalent can be found, take advantage of it. Otherwise, stick with existing LSST defaults.",10 DM-6816,"Process HSC ""RC"" dataset through the LSST stack","Process the ""RC"" dataset used to verify HSC data releases through the LSST stack using the configuration specified by DM-6815.",4 DM-6817,"Compare HSC and LSST processing of RC dataset","Using the script enhanced in DM-6588, compare HSC and LSST (DM-6816) processing of the RC dataset.",8 DM-6818,"Quality check LSST processing of RC dataset","Perform a quality analysis on the LSST processing of the RC dataset (DM-6816) in the same way as would be performed before an HSC data release.",5 DM-6819,"Resolve CModel issues with aperture corrections","While working on DM-4202 it became apparent that the aperture corrections calculated and applied by CModel were too large. This ticket is intended to trace down where the failure is occurring and correct it.",4 DM-6820,"Develop resource loaded plan for executing DRP sections of LDM-151",NULL,20 DM-6821,"Deliver DRP slide deck for LSST Director's Review","Required by 2016-07-12.",10 DM-6822,"Add meas_extensions_ngmix to lsst_distrib","Primarily so it can enjoy the benefits of regular CI runs.",2 DM-6824,"Use meas.algorithms.astrometrySourceSelector in measOptimisticB","Now that there is a working astrometrySourceSelector (just merged in meas_algorithms from DM-5933), we should get matchOptimisticB working with it. This would entail replacing matchOptimisticB.SourceInfo with AstrometrySourceSelectorTask and tweaking the latter to do whatever matchOptimisticB needs, and removing SourceInfo.",2 DM-6828,"Deliver sections for Operations Use Case Report","For each diagram covering a key use case, provide a narrative interpretation of the key concepts being conveyed, including significant operational implications from the concepts being presented. Fill in the table for the assigned use case areas. Key use cases/concepts include: L1 production, L2 production, ITC incident response, ITC problem management.",6 DM-6829,"Deliver sections for Concept of Operations","Contribute to Concept of Operations sections about Chilean, NCSA, and CC-IN2P3 facilities. Describing the ""nuts and bolts"" basics and summarize each facility's role in the LSST operational system.",3 DM-6830,"Investigate effects of turning on the Brighter-Fatter correction for single-frame processing of HSC data","In the process of comparing HSC vs. LSST stack single-frame processing runs, we have been running with the Brighter-Fatter correction (BFC) turned off. The reason for this to begin with was that is was not yet available on the LSST stack when we started these comparisons. We also want to isolate as many features as possible in order to confidently assess their individual effects The functionality was ported on DM-4837 with a default of *doBrighterFatter=False*. This issue is to continue the single-visit run comparisons (see e.g. DM-5301, DM-6490, DM-6491) with BFC turned on on both stacks. In particular, we are finding that slight differences in the reference stars selected for a given CCD can result in significantly different psf models. Also, it was noted in DM-4960 that LSST seems to select reference stars to a brighter cutoff than HSC. If a given field has a larger fraction of bright stars considered in the psf modeling, it is conceivable that it will be more significantly influenced by the BF effect, thus causing the large CCD-to-CCD variations seen in, e.g. DM-6490 (https://jira.lsstcorp.org/secure/attachment/28213/compareVisit-v1322-diff_base_PsfFlux-skyZp.png).",6 DM-6831,"Wrap base with pybind11","Split off from DM-6302.",2 DM-6832,"Wrap utils with pybind11","Split off from DM-6302.",2 DM-6833,"add 'placeholder' attribute to the input element","An attribute called placholder is available in html element <input> to give a hint to the user of what can be entered. The placeholder text must not contain carriage returns or line-feeds. Add it as proptype to <inputfield> component.",1 DM-6834,"Write report on SPIE conference","Write a report on my visit to the SPIE conference in Edinburgh.",3 DM-6835,"Learning about Openstack","Sahand progress on learning about Openstack",4 DM-6836,"Create a python interface to access OpenStack","Sahand progress on getting the interface to access the Openstack interface using Nova Client",3 DM-6837,"Data Backbone Conops iteration 1 prep: Create a list of service endpoints","Create a for list of service endpoints, with service considerations, and deliver to the Development file tree. Del with new ambiguities from the camera meeting at SLAC by listing the ""summit data services"" for both main camera and spectrograph as service endpoints, since this may increase the functionality required, and it seems prudent to flow any of these requirement into further processes, since they seem likely.",2 DM-6838,"Learning about Spark","Sahand progress on getting familiar with Spark and use of the interface to create a small Spark cluster in OpenStack",4 DM-6839,"Learning about Docker","Sahand progress on learning Docker containers and potential automatic deploy in OpenStack",3 DM-6840,"Set up and install Spark","Sahand progress on getting Spark installed ",2 DM-6841,"Learning about Kubernetes","Sahand progress on learning about automatic deploy and scalability of Docker containers using Kubernetes ",2 DM-6842,"Deal with emergent related requests affecting operations planning in June","There emergent request for comment emerged in June. 1) The interim project manager, directed that the project begin an investigation into Amazon Wen Service due to contacts he developed at a Data base orient workshop he sponsors. Formulating a response required a review of the service offered by AWS, and inquiring about the validity of pursing an evaluation of just one vendor in a marketplace that has many vendors, and a deciding that an appropriate amount of work was to send additional NCSA staff to an AWS workshop to gain a similar appreciation of AWS as was gained at the database meeting at SLAC. (Authority of interim project manager to insist on immediate action was also sorted out) 2) Request to understand computing capabilities at alternate site from the Deputy director. Support for for alternate site capabilities are documented in the the emerging L2 Batch concept of operations a copy of which was shared (though draft status noted) 3) Processed a summary of the Camera meeting which occurred at SLAC. Did not find conclusions that related to a concept of operations. IN particular we could not understand it there was a call for computing and a summit data service to support disconnected operations, or if this was a mere optimization in the system to relocate the acquisition and forwarding infrstructure to the summit, with no other changes.",4 DM-6843,"Learning about Swift and HDFS","Sahand progress on storage objects to be used in OpenStack",4 DM-6844,"Learning about Openstack and Jupyter","Di progress on learning these web technologies",5 DM-6845,"Installing JS9 in Openstack server",NULL,2 DM-6846,"Learning about SocketIO and HTML REST API","Di progress on Communication technologies for the web",4 DM-6847,"Integrating Jupyter and JS9 for FITS visualization","Di progress in getting JS9 to work in Jupyter notebook",6 DM-6848,"Write wrapper API for JS9 and Jupyter","Di progress in writing a wrapper to interact between JS9 within Jupyter",5 DM-6849,"Understand the installation and administrative processes","Review and gain administrative insight using the processes encoded in test scripts and other relevant features based on investigations in the prototype installations. For example, we may observe steps in test scripts to gain understanding of capabilities behind the scripts. Provide comments on documentation where deemed helpful.",9 DM-6850,"Liaison with deployment effort","Interact with qserv developers supporting deployment and NCSA's service provisioning environment. Learn and investigate aspects of qserv administration present in test deployment but not previously covered.",10 DM-6851,"Setup multinode test environment for initial learning about installations ","Setup up one master node and one worker node.",3 DM-6852,"Update Activator to reflect recent changes in CmdLineTask",NULL,4 DM-6853,"Discussion regarding 'quanta' definition in SuperTask",NULL,2 DM-6854,"Finalize documentation and current issues of prototype","After updating some latest changes, need to update documentation to explain the extend of this supertask and activator initial implementation.",4 DM-6855,"TBD related emergent work in July",NULL,5 DM-6856,"TBD related emergent work in August",NULL,5 DM-6857,"Document that the catalog returned from star selectors is a view","Star selectors return a catalog whose records are shallow copies of the input catalog records. Document the shallow copy aspect. This is important for two reasons: - The user should know - Implementers must be told this, because if the records are deep copies then the code that sets a flag for stars will not set a flag in the input catalog, which loses most of the point of setting that flag.",1 DM-6858,"Mapper tests require modification when new datasets are added","[~price] [recommends|https://community.lsst.org/t/centrally-defined-butler-datasets/841] a new way to define datasets common to all cameras in daf_butlerUtils, but modifying these yaml files require explicit lists of datasets to be modified in tests/cameraMapper.py. If these tests are still useful, they need to depend on a minimal set of dataset definitions instead of the real ones.",1 DM-6859,"Participation according to direction from interim project management","Given directions from interim project management, participation consisted of direct conversations with Kevin and Jacek plus background work talking to staff related to assembling a plan.",5 DM-6860,"Refine simple 1D DCR correction","DM-5695 created a functional implementation of a simple DCR correction algorithm. While it appears to successfully create template images with airmass and DCR matched to science images, it is computationally inefficient and appears to introduce new artifacts to the template image. This ticket is to enhance the simple algorithm in several ways: * Convert to sparse matrices where possible * use variance weighting of the images * propagate masked pixels correctly * Refine the algorithm to mitigate the new artifacts",6 DM-6861,"Understand how to render conops documents in Sphinx","Learn how to render conops documents in reStructuredText. Prototype conops template and for delivery into Technical Control Team Sphinx engineering environment.",1 DM-6862,"Raw draft of System Monitor and Comfort Display","Produce raw draft of conops for review by steering committee. Includes operational components and connectivity for the system monitoring services that will monitor devices from the summit to NCSA.",3 DM-6863,"Add verification feature to L1 conops",NULL,7 DM-6864,"Add verification test to L1 plan",NULL,7 DM-6865,"Add verification test to L1 design",NULL,7 DM-6866,"Add verification feature to Data Backbone conops",NULL,10 DM-6867,"Add verification feature to L2 conops",NULL,7 DM-6868,"Add verification feature to Authentication & Authorization conops",NULL,10 DM-6869,"Liaison with Systems Engineering",NULL,6 DM-6870,"Appreciate amount of effort needed to run preliminary planning exercise","Run planning process with local staff to appreciate amount of effort needed.",5 DM-6871,"Review evaluation criteria with CC-IN2P3","Review evaluation criteria with Fabio during his visit from CC-IN2P3 to NCSA. https://drive.google.com/open?id=1Xhj6kaFEnNhCyRPskB6BCXYxgfs_9-cl3BRX9wCh1sE",1 DM-6872,"Create evaluation plan from evaluation criteria","Turn criteria into tabular comparison chart and respect test implementation constraints.",4 DM-6873,"Estimate amount of effort needed to run detailed planning exercise","Run through process of detailing activities down to story size requested by the LSST EVM system. Do detailed estimation of conops development and a sample of technical areas, and extrapolated based on number of epics, size of staff, and complexity of mission. Total = 100 hours for 3 months of activities for current staff size.",2 DM-6874,"Design framework for integrating procurement activities with invoices","Respond to request to relate equipment charges to acquisition strategy document procurement activities.",1 DM-6875,"Design framework for reporting and steering meetings","Run the process with staff to assess and supervise technical status, the appropriateness of work compared to architectural vision, consistency with NCSA general acumen, and status vs. plan.",6 DM-6876,"TBD processes coordinated with impending hire","Design and implement critical processes defined in the RACI document, coordinated with impending hire.",6 DM-6879,"Address concerns with source side (Dave Mills)","Work with Dave Mills and others to understand architecture and use of ""source"" EFD. The goal is to understand the amount of volume of data that would be in reformatted EFD that otherwise would not have been, should we proceed with the proposed change.",8 DM-6880,"Address concerns with target side (SLAC)","Understand permissions and protections that would be in the reformatted EFD that otherwise would not have been, should we proceed with the proposed change.",8 DM-6881,"Address internal concerns","Understand whether the file annex should be kept in the same cluster as EFD as opposed to general files in the data backbone.",6 DM-6882,"Incorporate into ConOps and any draft design notes","Incorporate concerns, solutions and agreements into ConOps and any draft design notes.",6 DM-6883,"Address additional emergent concerns ","Address TBD additional emergent concerns ",2 DM-6884,"Rework MemMan to be inline with the qserv worker Scheduler.","Split the memory mapping function from the memory locking function to allow the scheduler to initiate locking without blocking. Add additional memory tracking improvements in line with current thinking. Reduce lock contention. Add logging.",4 DM-6886,"forcedPhotCoadd.py fails on CFHT data due to a CModel bug","Hello, forcedPhotCoadd fails while running on CFHT data due to a CModel bug. Here is an example on the error message that we get: {code} python: src/CModel.cc:1368: void lsst::meas::modelfit::CModelAlgorithm::measure(lsst::afw::table::SourceRecord&, const lsst::afw::image::Exposure<float>&, const lsst::afw::table::SourceRecord&) const: Assertion `measRecord.getFootprint()->getArea()' failed. Aborted {code} Adding the following lines in cmodel.py (in CModelForcedPlugin.measure, before the call to self.algorithm.measure) allows to go around the problem for the time being, which seems to arise for null value of the number of pixel in a given footprint: {code} if not measRecord.getFootprint().getArea(): raise ValueError(""measRecord.getFootprint().getArea(): 0. No pixel in this footprint."") {code}",1 DM-6890,"deploy jenkins python env support",NULL,1 DM-6892,"Access to system with LSST stack","Secure access to machine(s) with the LSST stack. This includes installation on local desktop/laptop.",2 DM-6893,"Controlled Test of LMSimpleShape using high SNR objects","Some issues came up during DM-6300 which indicated that a more controlled set of tests would be required than the random Great3Sims tests to understand the behavior of NGMIX LMSimpleShape. LMSimpleShape appears to fail computing moments on low SNR objects. It also shows pretty wide variation in shear bias which did not show up with CModel. The needed tests with would include controlled profiles (Gauss, Dev, and Exp), controlled SNR, and controlled q, theta, and flux. This should separate out the causes of failure and shear variation which we have seen.",6 DM-6894,"Ensure DipoleFitTask uses correct PSF(s) in case when Decorrelation is turned on","Diffim A&L decorrelation (DM-6241) modifies the diffim PSF, but leaves the ""pre-subtraction"" images used by DipoleFitTask as they were. Ensure that the correct PSFs are being used for dipole fitting when decorrelation is turned on (and actually, in all cases).",8 DM-6897,"Get data stream from socket into a fits file","Get data stream (module?) from Jim into a fits file than can be loaded subsequently to the butler.",2 DM-6898,"Load known image data format into the Butler","Use some type of known data (image) to load and test into the buttler. Data types might include DECam MEF images of single-plain image files from simulations.",2 DM-6899,"Assemble data stream from socket to lsst-stack pipeline","Connect all of the parts together.",4 DM-6900,"ci_hsc failure: insufficient PSF sources classified as stars","Since [ci_hsc#396|https://ci.lsst.codes/job/ci_hsc/396/], the regular ci_hsc build has been failing with: {code} [2016-07-05T23:59:53.929169Z] FATAL: At least 95% of sources used to build the PSF are classified as stars (49 > 50): FAIL [2016-07-05T23:59:53.929201Z] Traceback (most recent call last): [2016-07-05T23:59:53.929238Z] File ""/home/build0/lsstsw/build/ci_hsc/bin/validate.py"", line 3, in <module> [2016-07-05T23:59:53.929249Z] main() [2016-07-05T23:59:53.929317Z] File ""/home/build0/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py"", line 53, in main [2016-07-05T23:59:53.929334Z] validator.run(dataId) [2016-07-05T23:59:53.929375Z] File ""/home/build0/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py"", line 163, in run [2016-07-05T23:59:53.929394Z] self.validateSources(dataId) [2016-07-05T23:59:53.929436Z] File ""/home/build0/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py"", line 201, in validateSources [2016-07-05T23:59:53.929451Z] 0.95*psfStars.sum() [2016-07-05T23:59:53.929510Z] File ""/home/build0/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py"", line 91, in assertGreater [2016-07-05T23:59:53.929547Z] self.assertTrue(description + "" (%d > %d)"" % (num1, num2), num1 > num2) [2016-07-05T23:59:53.929587Z] File ""/home/build0/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py"", line 82, in assertTrue [2016-07-05T23:59:53.929614Z] raise AssertionError(""Failed test: %s"" % description) [2016-07-05T23:59:53.929660Z] AssertionError: Failed test: At least 95% of sources used to build the PSF are classified as stars (49 > 50) {code} This error appears to be related to DM-5877: when I reverted to versions of pipe_tasks, meas_base, meas_algorithms, ip_diffim, meas_extensions_photometryKron and obs_subaru predating that ticket landing, the error vanishes. I note that [~nlust] reports that he does not see these failures on his (OS X) system, but I can reproduce them on Linux: we should investigate if that's just a coincidence, or if there is per-platform variation here.",1 DM-6902,"VO search doesn't trigger coverage image nor overlay","While migrating the VO search panel, i found the follwing problem: once the table gets back, no image coverage or overlay is rendered. One problem from IRSA simple cone search result is that the VO table doesn't contain the right UCDs expected. The second problem when the VO table does contain the proper UCDs is that the META_INFO is not set. The edu.caltech.ipac.firefly.server.query.SearchManager#jsonTablePartRequest doesn't set the attributes from the DataGroup object into the TableMeta data as it is done previously in OPS by edu.caltech.ipac.firefly.server.query.SearchManager#getRawDataSet Please, add the META_INFO object missing to the table and set the proper CATALOG_OVERLAY_TYPE, and CATALOG_COORD_COLS needed so the Coverage image and overlay can be rendered.",4 DM-6903,"Add an option to label ccd serial number on the showVisitSkyMap.py plot ","(actual assignee: Samuel Piehl) Sometimes it is useful to know where the CCDs are on the plot. Add an option to label the CCD numbers. ",1 DM-6904,"Create DCR visualization tools","Several visualization tools will be very helpful to fully understand the effect of DCR correction algorithms and their failure modes. * A function that generates difference images with the sources used for calibration and/or psf fitting marked. * A visualization that indicates the spectral type of each source in an image. This could be a mask overlay where the color corresponds to the spectral type. * A visualization of the coarse spectral resolution model built for DCR correction",5 DM-6905,"Locate the test dataset for PDAC","Locate and evaluate a dataset of SDSS Stripe82 which is going to be used for testing the prototype DAC.",2 DM-6907,"XYPlot density plot with log scale - bin size is not reflected correctly","XYPlot with the large number of points does not display correctly when log scale is selected. When log scale is selected, binning on the server should be using the logs, so that the bins are the same size on the log scale. ",4 DM-6908,"Filter editor on a chart toolbar","Need to add filter editor to the chart toolbar. Filter editor should be without selectable rows.",4 DM-6909,"Filtering from expanded mode cancels expanded mode","When a table is filtered from the expanded mode, the layout is changed back to unexpanded. It looks like the issue is more general: table actions trigger layout changes, which are not always right. For example, TABLE_REMOVE action while in a dropdown makes the dropdown to get closed. I've traced it to FireflyLayoutManager.js:layoutManager generator function. Test sequence in firefly: - When a table is loaded, open ""Charts"" dropdown, select Col link for X, then select Col link for Y. (At this point the previous table is removed). - TABLE_REMOVE action on the second click triggers dropdown to go away. ",2 DM-6914,"git-lfs.lsst.codes certificate is expired","Per reports on hipchat, the tls certifcate on git-lfs.lsst.codes was not upgraded to the new *.lsst.codes cert. {code:java} John Swinbank 9:52 AM @josh @jmatt I'm seeing the following, which I think might be the same as @srp's error above. Any ideas? Get https://git-lfs.lsst.codes/objects/24874b686b9479a823987dc2bd2700cad5b73e74a43108fb61b91d7f79f0cd99: x509: certificate has expired or is not yet valid Followed by git lfs failing. (I assumed it was user error on my part at first, but if so it's coincidence that Steve's git lfs fails at the same time.) {code} ",1 DM-6915,"jointcalRunner passing tract to jointcal, which had tract removed from run()","When cleaning up jointcal for testing, I removed tract from jointcal.run(), but did not remove it from the return list of JointcalRunner.getTargetList(). Tract isn't actually used anywhere in jointcal.run(), so we should be able to just remove it from getTargetList's return. Keeping those two in sync may be a bit tricky without a unittest that compares them.",4 DM-6916,"Documenteer seeds Git revision date and branch name if not present in metadata.yaml","If {{last_revised}} and {{version}} are not present in metadata.yaml, then the Git commit date and branch name should be used while building metadata instead. Also updates lsst-technote-bootstrap to take advantage of automated metadata for new projects.",1 DM-6917,"Write User Guide for new validate_drp metric/measurement API","DM-6629 provided a new API for consistently specifying metrics, their specification, and reporting results of measurements. This API can, and should, be used beyond validate_drp for any code that wants to submit metadata to SQUASH. This ticket will provide user documentation on the API base classes to help other developers write new metrics and measurements.",4 DM-6918,"Implement script to simulate AP workflow","To understand better the load on L1 database I need a more or less adequate set of queries running against the databases. AP-generated queries should be a good start so a simple script that simulates what AP does will be very helpful. Sure I don't need any actual image processing or alert production, only the parts which read/write data to the database on per-visit basis.",10 DM-6919,"Please rename ""afterburners""","In DM-4887 we introduced a new measurement post-processing system which we called ""afterburners"". The term ""afterburner"" is overloaded and applied in multiple contexts. To save confusion, please rename this system to something less ambiguous. Best if we can do this soon, before this usage spreads.",1 DM-6922,"Upgrade to new stack install procedure for containers","LSST stack install has evolved: https://pipelines.lsst.io/install/newinstall.html# Release container creation script needs to be update. Latest Docker version will be tested, as [~bvan] reported cmd line options have changed.",2 DM-6923,"Apply distortion when searching for astrometric reference objects","While investigating DM-6529 I found that LSST generally finds fewer reference objects than HSC when doing astrometry. For the CCDs on the edge of the focal plane the number of stars was typically very low causing frequent failures. I found that in the HSC code, there is a distortion being applied that shifts the exposure bounding box when getting objects from the reference catalog. This distortion is not being applied in the LSST code.",1 DM-6924,"Resurrect obs_file","obs_file needs to be resurrected. This is partially due to the reorganization of processCcd. My take is to try to make the ingest script read the files and ingest them keyed on the filename. Then the dataId will be just the filename. Hopefully we can then mock all the other info needed for processing in a general way. Calibration (astrometric and photometric will be off by default). ",8 DM-6925,"star selector and PSF determiner are selecting stars that are not valid point sources","When turning on CModel a more robust extendedness classifier relieved that many of the stars being used as PSF candidates were being classified as extended as shown in the attached plot. This plot was generated from the output of ci_hsc. Work should be done to determine why these stars are mistakenly being selected and fix the bad behavior. Additionally the [temporary work around in ci_hsc|https://github.com/lsst/ci_hsc/commit/6daf43ca41b6d192b6e1dbedb60cde0bec90b615], where the success criteria for validate sources in validate.py should be reverted from 85% to 95%.",4 DM-6928,"HSC backport: Include PSF moments in the output tables","This is effectively a port of [HSC-110|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-110] though, due to the considerable differences in bookkeeping for the SdssShape code, this will be more of a reimplementation. ",4 DM-6933,"access policy for PDAC","This is a Prototype DAC (PDAC) and the access to it is limited. We need to draft a access policy. ",1 DM-6934,"Startup Scaffolding Machine Requirements - Base","Plan Base machine needs for Startup, including those processes that must run together on the same physical host (such as databases resident in memory needed by other processes).",2 DM-6935,"Plan Base site machine startup pattern","List all dependencies for specific processes that must be previously up an running. Establish the settings for 'time zero' on the startup timeline, such as purging queues, clearing specific data stores, arranging file system for startup, etc.",2 DM-6936,"Prep for and attend Kubernetes meeting","Kubernetes is a machine provisioning application that has potential to assist LSST Startup scaffolding such as Just In Time personality assignment, persisting config settings, etc.",1 DM-6937,"Download and install Kubernetes","Implement a Kubernetes instance running on the Nebula cluster and begin configuration for testing Base site Startup behavior.",4 DM-6938,"Evaluation of Kubernetes for Startup Scaffolding","After Kubernetes is running as a simulated base site start up and provisioning tool, begin evaluation with fault injection such as the need for hot swap machine failover, sudden changes to Network topology and name server entries, etc.",3 DM-6939,"Set up proposed start up tools and procedure for NCSA L1 components","If Kubernetes is the answer for startup provisioning which it is hoped to be, apply it to NCSA L1 machine startup requirements.",2 DM-6940,"Final Startup Scaffolding document","This is expected to be a document specifying the final Startup scaffolding disposition. If specification is not final, it will assess which requirements for this epic were not reached.",2 DM-6941,"First round of updates to DRP LDM-151 sections from reviews","Will address comments from [~swinbank], [~rhl], and probably [~ktl].",4 DM-6942,"Explore and experiment the process of creating a Jupyter widget","Study the Jupyter notebook and understand the concept of Jupyter widget. Try to make a simple Jupyter widget that works with Firefly visualization.",4 DM-6944,"Integrate multiple-backgrounds concept into LDM-151","It's recently become apparent that we need to at least consider using different background estimation techniques for different kinds of measurements. This is will require some thought to work into our current processing plans.",4 DM-6945,"Add text to algorithmic components sections in LDM-151","While [~swinbank] has commented that the outlines are probably good enough for planning work (and I thnk that's broadly true), the lack of text in the algorithmic components section did occasionally lead to some misunderstandings in [~rhl]'s first review pass, so I think I should flesh that out with text sooner rather than later. In this issue, I'll stick to sections that no one else has added text for, but eventually I'll also need to work with [~krughoff] and perhaps others to ensure that section has a consistent level of detail and focus.",8 DM-6949,"Firefly has problem to render in other browsers than Chrome","Couple of problem using Firefly in Safari: * the components appears blank, in Firefox: * image and xyplot are not aligned (Gator). The alignment can be reproduced in my Chrome and Safari. Search parameters: ALLWISE source catalog, m81 100arcsec. ",1 DM-6952,"Table problems","Table component has couple of problems: # Scrambled table values after column selection and saving the table, then reset mess up the table. Saving the table and reset makes the table comes back. # table display no longer redefines column names in the table based on the column label, e.g. ""Field Size"" instead of ""s_fov"". # Downloaded file is not a valid IPAC table. # Filtering table does not change image overlay or plot until table is saved. At that point, the filtering works, but the plot symbol changes (happens when result is decimated, datapoints > 5000?) # In Edit Table Options, it's unclear what the reset button resets to. ",6 DM-6953,"Image problems are grouped in this ticket","Image viewer has a couple of issues: * There is no panner for the image (is it missing from the API or is it a bug in calling it?) * No readout value from thumbnail image * -Image does not have toolbar or layers control (Gator) - probably API options to be used or missing?- -> move to DM-7001 * -Markers don't show up in PNG download- -> moved to DM-6980 * The top bar readout doesn't include units for the pixel flux * Clicking on the expand icon deletes the image (in API only) * Clicking on the expand icon diabled the expand mode of table and xy plot (in API only) * Image button ask you for a position and displays that, wiping out whatever image brought you to IRSA Viewer in the first place. It should give another tile of the same field, drawn from the selected data set. (IRSAVIewer only) * Need mode to draw grid without labels * RangeValues messing up image display {noformat} old: var external= firefly.getExternalViewer();             external.setDefaultParams({   ""TitleOptions"" : ""FILE_NAME"",                                           ""ColorTable""   : ""1"",                                           ""TitleFilenameModePfx"" : ""cutout"",                                           ""PostTitle""    :  ""\locstr\"",                                           ""OverlayPosition""    :  ""\lon\;\lat\;EQ_J2000"",                                           ""RangeValues""  : firefly.serializeRangeValues(""Sigma"",-2,8,""Linear"")                                       }) ; new: xtViewer.setDefaultParams({  ""TitleOptions"" : ""FILE_NAME"",                                           ""ColorTable""   : ""1"",                                           ""TitleFilenameModePfx"" : ""cutout"",                                           ""PostTitle""    :  ""\locstr\"",                                           ""OverlayPosition""    :  ""\lon\;\lat\;EQ_J2000"",                                       }) ; The ""RangeValues"" were taken out for now ""RangeValues""  : firefly.util.image.serializeSimpleRangeValues(""Sigma"",-2,8,""Linear"") {noformat} ",14 DM-6954,"XY plot problems found","implement the items listed here: * min/max options are now gone after migration, need to be added. * Use the expression for X, Y column as the default label, otherwise the read out could be confusing. * label changes for decimation: X-Bins and Y-Bins: Number of X-Bins, Number of Y-Bins * shrink the size (to 1/5?) of the blue dots for data representation. I do like the circle when the point is highlighted. * The units on the plot are indicated with a comma, e.g. “dec, deg”, should be ""dec (deg)"" as before Need to confirm again ([~ejoliet]) * -Making a change to the plot (e.g. ra = ra * -1), then clicking on the gears to close makes the shading legend disappear. It also “quantizes” the plot- (not any more) * What happens now is the plot appears without legend after a search. Then making a change to the plot (e.g. ra = ra * -1 or filtering the table), then applying makes the legend appears/disappears. Expanding the table and collapsing it, make the legend disappears. *- Step to reproduce: Catalog search on 2MASS around m16 with 10' radius. * Greyscale introduced, where different colors represent different numbers of points. After filtering, the points change color to blue. (This is because the it is not decimated any more) * -Clicking on plot gears makes plot unusably small.- (Not applicable any more since gear now brings up the options in popup) ",6 DM-6955,"Message Dictionary Adjustment.","Audit format of existing messaging and adjust according to 'wants' not task 'needs'...that is, msg body format that exists now is sufficient to fulfill tasks, but destination components must receive a broader description of overall system state. This will allow all components to log a more comprehensive snapshot of current state and is needed for troubleshooting. These additions to the message dictionary will be configurable like a logging priority levels function, and additions to message payload can be turned off for typical nightly operation.",4 DM-6956,"'ACK' (Acknowledgement) message formats","Enumerate ACK messages for all primary message types. Prototype both blocking and non-blocking acknowledgement aggregator that works via timeout, behavioral change, etc. This is related to DM-6411",4 DM-6957,"Adding ACK messages to existing code framework","New entries in message dictionary must be added and tested with the existing messaging code base.",4 DM-6958,"Documentation for new message types","Add new ACK message types to existing Dictionary documentation.",2 DM-6959,"Messages as objects","Consider creating a dictionary of code objects to represent messages. Currently message bodys are built on the fly - evaluate pros and cons of switching to a message factory pattern. Message types are not a very extensive list, but switching to object implementation could increase maintainability of code.",2 DM-6960,"Overlay health check code","Implement and overlay health check mechanism on existing messaging control code. Prototype and gather timing information to determine optimal frequency of checks and the location in the exposure cycle when these checks should occur.",8 DM-6961,"Fault Injection in the form of unsuccessful health checks for components","Build testing mechanism to inject faults into into health framework and stub code to address health check failures",4 DM-6962,"Create policy for health check failure","Document policy regarding action to take when various components are found to be unhealthy. This can vary depending when 1 component type (Forwarder) is offline versus 21 Forwarders offline. In addition, plans must be formulated for addressing the point in the exposure cycle when the health failure occurs.",4 DM-6963,"Implement health failure policy","Formalize the prototypical implementation of health checks and associated policy into 'what to do' actions",4 DM-6964,"Make a proposal for API support for representation of relationships between table columns","End users and the SUIT need to be able to determine a variety of relationships between columns in the tabular data products produced by LSST. The particular example motivating this ticket is the need to answer the question ""where in the table is the uncertainty data for column 'x'?"". The answer could be: * ""there isn't any"" * ""a symmetric Gaussian uncertainty is in column 'sigma_x'"" * ""asymmetric Gaussian uncertainties are in columns 'sigmaplus_x' and 'sigmaminus_x'"" * ""'x' is correlated with 'y' and the covariance matrix is in 'covar_xx', 'covar_xy', and 'covar_yy'"" Ideally we would find a way for these relationships to be defined when the Apps code generates its afw.table outputs, discoverable through an API usable in the afw.table context, exportable to the database, and made available to end users and the SUIT. It should be usable whether the data are delivered to end users as reconstituted afw.table objects or as tables in common Python formats (at least Astropy tables). It should assist the SUIT in determining how to (automatically, though optionally) display uncertainty data when the primary data are requested. This ticket expresses the idea that a solution that consists purely of a documented convention about prefixes to the string names of columns is inadequate. We would like to avoid having to write code implementing that convention in, potentially, hundreds of places, and we would like to avoid requiring that end users know these conventions in order to see proper displays with error bars. ",3 DM-6966,"Flesh out software primitives","Jim has put together a fairly complete software primitives section. This task is to read it over from the perspective of Alert Production and expand/refine where necessary.",4 DM-6967,"Level 2 conops formatting: convert draft to reStructedText",NULL,1 DM-6968,"create a shared stack on NFS for use with the current local condor pool","It is well known that building, setting up a stack, and interactive devel work with those operations on NFS has performance issues. Hence the official shared stack on lsstdev uses /ssd . However, a shared stack on NFS is useful and adequate for one important use case -- users need a stack that can be used for small productions on the local condor pool currently available on lsstdev. For this use case multiple ""source""/""setups"" on a node/against the file system can be avoidable by using a script to directly declare the environment. run_orca /ctrl_orca supports this feature. While GPFS is coming soon, there is expected to be a transition period of 2-3 months and so the NFS file system and a stack on it can serve users for an interim period. If building a shared stack on NFS is not a heavy labor, we think it is worth the effort for this interim period, and as such make this request for a shared stack on NFS. ",1 DM-6969,"Fixes to LoadIndexedReferenceObjects","Bug fixes for using the new {{LoadIndexedReferenceObjectTask}} and its associated components.",2 DM-6970,"Add tests for bindings of Eigen::Array and ndarray::EigenView","[~pschella] has discovered that we don't have test coverage for converting less-common Eigen types to Python. This is not urgent, but it should be fixed.",1 DM-6971,"Qserv 2016_07 release","- Update release notes - Publish docs and bump version numbers",1 DM-6972,"Fix Qserv install doc and scripts for new newinstall.sh","Update qserv install docs per new info at https://pipelines.lsst.io/install/newinstall.html",1 DM-6973,"Fix metadata date problem in LDM-{463,152,135}","These docs are currently borken in CI -- just need to have the dates reformatted in their metadata.yaml",1 DM-6974,"Type of IngestIndexedReferenceTask_config wrong in obs_ paf files","In DM-6651 I moved the new HTM indexed reference catalog code from pipe_tasks to meas_algorithms, but didn't do a complete job. The type of IngestIndexedReferenceTask_config in obs_ paf files still must be updated.",1 DM-6975,"Document release milestone changes in DMTN-020","Please add a note to DMTN-020 describing the changes to release milestones discussed at the [DMLT meeting of 2016-07-18|https://confluence.lsstcorp.org/display/DM/DM+Leadership+Team+Meeting+2016-07-18].",1 DM-6976,"watch for Highcharts update ","There is an issue in the density plot for displaying the legends. Highcharts does not support the setting of the symbol size in the legends. So when the symbol size is too small or too large, the legends are not displayed. We don't want to do too much workaround currently. This ticket is to watch for the Highcharts update. ",1 DM-6977,"verification and test of the Bayesian histogram calculation on server side","We need to set up some unit tests of the Bayesian histogram calculation. First we need to do some verification of our algorithm with scientists. I think we can find some known data sets and the results with help from scientists. The unit tests should include several different distribution of input data. ",10 DM-6978,"Update qserv for changes in Log interface","DM-6521 improved Log class interface by replacing some static methods with non-static. Qserv is currently using couple of static methods which were retained in Log class for the duration of this migration. Once updated log package is released update qserv code to use new non-static methods and remove static methods from Log class after that.",2 DM-6980,"Markers don't show up in PNG download","Markers don't show up in PNG download",4 DM-6982,"Fix oversampling settings in psfex","The current settings in psfex will only turn on oversampling only if the seeing is < 0.5"", even if you have configured it do oversampling. This needs to be changed so that everything is determined by the config parameters. We have also seen on HSC data that oversampling in general does not work well in psfex. We need to change the current configuration which does 2x oversampling to just use the native pixel scale.",1 DM-6983,"ci_hsc failure: AttributeError: 'Butler' object has no attribute 'repository'","Following [~npease]'s [recent changes to the Butler|https://community.lsst.org/t/im-checking-in-butler-changes-related-to-rfc-184/959], ci_hsc is failing as follows: {code} [2016-07-20T07:57:31.954576Z] Traceback (most recent call last): [2016-07-20T07:57:31.954643Z] File ""/home/jenkins-slave/workspace/stack-os-matrix/compiler/gcc/label/centos-7/python/py2/lsstsw/build/ci_hsc/bin/validate.py"", line 3, in <module> [2016-07-20T07:57:31.954664Z] main() [2016-07-20T07:57:31.954732Z] File ""/home/jenkins-slave/workspace/stack-os-matrix/compiler/gcc/label/centos-7/python/py2/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py"", line 53, in main [2016-07-20T07:57:31.954756Z] validator.run(dataId) [2016-07-20T07:57:31.954825Z] File ""/home/jenkins-slave/workspace/stack-os-matrix/compiler/gcc/label/centos-7/python/py2/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py"", line 155, in run [2016-07-20T07:57:31.954851Z] self.validateDataset(dataId, ds) [2016-07-20T07:57:31.954923Z] File ""/home/jenkins-slave/workspace/stack-os-matrix/compiler/gcc/label/centos-7/python/py2/lsstsw/build/ci_hsc/python/lsst/ci/hsc/validate.py"", line 117, in validateDataset [2016-07-20T07:57:31.954956Z] mappers = self.butler.repository.mappers() [2016-07-20T07:57:31.954991Z] AttributeError: 'Butler' object has no attribute 'repository' [2016-07-20T07:57:32.023212Z] scons: *** [.scons/ingestValidation-903342-100] Error 1 {code} See e.g. https://ci.lsst.codes/job/stack-os-matrix/13274/compiler=gcc,label=centos-7,python=py2/console. Please fix it. ",1 DM-6984,"Suggest logging migration in daf_persistence and daf_butlerUtils","Use lsst::log instead of pex::logging in daf_persistence and daf_butlerUtils",2 DM-6985,"Suggest logging migration in afw","Suggest a changeset with lsst::log instead of pex::logging in afw",5 DM-6986,"Suggest logging migration in pipe_tasks and meas packages","Suggest changesets using lsst::log instead of pex::logging",5 DM-6987,"Write up a description of Composite Datasets based on input from KT","Write a description of composite datasets as I understand them based on the email KT sent on May 20 (attached), and on conversation I had with KT and Fritz on July 20.",2 DM-6988,"Review Composite Dataset description document with stakeholders","Review the composite dataset description document with [~jbosch] and [~Parejkoj], and any others who may be interested (e.g. post on community or do an RFD)",2 DM-6989,"ctrl_events/tests/EventAppenderTest.py fails Jenkins run-rebuild","ctrl_events/tests/EventAppenderTest.py started failing on Jenkins ""run-rebuild"" last night: https://ci.lsst.codes/job/run-rebuild/354//console All test cases in EventAppenderTest.py did run and pass, but it failed with a Segmentation fault in the end. Jenkins ""run-rebuild"" uses a stack on NFS on lsst-dev (/nfs/home/lsstsw). The same test passes on regular Jenkins (stack-os-matrix). ",3 DM-6990,"Improve testing in SQuaSH prototype","This ticket captures some testing practices from https://ep2013.europython.eu/media/conference/slides/obey-the-testing-goat-rigorous-tdd-for-web-development-with-django-and-selenium.pdf that we intend to use in the SQuaSH prototype. - Use selenium to test user interactions (functional tests) - Use unittest module for unit tests - Include some documentation about testing in sqr-009 ",5 DM-6991,"Add a script to summarize what visits are in what patches","(Actual assignee: Samuel Piehl) Have a script to show what visits are in what tracts/patches. This is especially useful for running coadd making and processing (e.g. makeCoaddTempExp, assembleCoadd) with runOrca and HTCondor, as the dataId of the jobs need to be specified. So this script's output will be in the format of a runOrca input file. ",5 DM-6992,"Extend SQuaSH dashboard to work with multiple datasets","Currently SQuaSH dashboard works only with a fixed dataset. We want to ingest measurements of metrics computed by validate_drp for multiple test data e.g CFHT, DECam and HSC. In order to handle multiple datasets, we need a new model in SQuaSH and extended the job API to ingest the measurements for different datasets. The user must be able to selected in the interface the dataset to be displayed.",3 DM-6996,"produce a draft document of SUIT requirements","After combing through the current SUIT requirements, we feel that we need to re-organize and re-write the SUIT requirements to be in-line with SUIT vision document. This story will be producing the first draft of the rewrite. ",20 DM-6998,"Problems with MemoryTest ordering","{{MemoryTestCase}} (or a derivative thereof) must be run as the last of all tests in a module in order to properly catch leaks. [Our documentation|https://developer.lsst.io/coding/python_testing.html#memory-and-file-descriptor-leak-testing] implies, and [SQR-012 states|https://sqr-012.lsst.io/#memory-test], that this can be achieved by listing it as the last test case in the file. This works for py.test, but not when using plain old unittest: the latter does not, so far as I can see, guarantee any sort of ordering as a matter of principle, and, in practice, it sorts things lexicographically (it uses whatever order it gets from running {{dir()}} on the test module, and I don't *think* that's guaranteed to be anything in particular). For example, consider [{{testAstrometrySourceSelector.py}}|https://github.com/lsst/meas_algorithms/blob/master/tests/testAstrometrySourceSelector.py]. I made the following change to introduce a memory leak: {code} --- a/tests/testAstrometrySourceSelector.py +++ b/tests/testAstrometrySourceSelector.py @@ -70,8 +70,9 @@ class TestAstrometrySourceSelector(lsst.utils.tests.TestCase): self.sourceSelector = sourceSelector.sourceSelectorRegistry['astrometry']() def tearDown(self): - del self.src - del self.sourceSelector + pass + #del self.src + #del self.sourceSelector def testSelectSources_good(self): for i in range(5): {code} Py.test catches it: {code} $ py.test-2.7 testAstrometrySourceSelector.py [...] testAstrometrySourceSelector.py .........F [...] {code} But simply running the test suite does not: {code} $ python testAstrometrySourceSelector.py .......... ---------------------------------------------------------------------- Ran 10 tests in 0.105s {code} Rename the test case: {code} @@ -144,7 +145,7 @@ def setup_module(module): lsst.utils.tests.init() -class MyMemoryTestCase(lsst.utils.tests.MemoryTestCase): +class xMyMemoryTestCase(lsst.utils.tests.MemoryTestCase): pass if __name__ == ""__main__"": {code} And boom: {code} $ python testAstrometrySourceSelector.py ......... 54 Objects leaked: {code} Based on a very quick check, I think [sconsUtils runs tests by simply invoking {{python}}|https://github.com/lsst/sconsUtils/blob/f9763768d999cefa4c26b9f3418c28394dfb38df/python/lsst/sconsUtils/tests.py#L133], and I'm pretty sure that this is hard-wired into the muscle memory of many developers. In these cases, memory tests written following current guidelines won't be being properly executed. ",3 DM-6999,"Use lsst::log in pipe_base and pipe_tasks","Per RFC-203, switch from using pex.logging to lsst.log in pipe_base and pipe_tasks (stage 2)",8 DM-7000,"Remove pex_logging dependency on pipe_tasks",NULL,3 DM-7001,"Gator / Image Vis issue","Issues with coverage: * Toolbar icon not showing up * If only one point that is no coverage image (or one table with many records but has the same ra,dec values) * CAN’T REPEAT: In expanded mode, magnifier fails when image fills the visible space entirely (seems to affect 'Coverage' image only) * Missing feature: before migration, in expanded mode, the toolbar had an 'added image' button which was bringing an image search panel to add images to the current view. => _move to:_ DM-7068 * CAN’T REPEAT: if marker/footprint overlay is clicked, that doesn't activate the image viewer and doesn't update the layer dialog either. * in laptop screen size, the toolbar is not fully visible, scrolling from left to right only move the background but not the expanded panel. * CAN”T REPEAT:in expand mode and zoom 'fill the visible space' clicked, the magnifier image doesn't show anything from the coverage image, can be reproduced in http://localhost:8080/firefly/demo/ffapi-highlevel-test.html (BTW, it happens in finderchart in OPS on any image in expanded mode ( ? ) ) * Readout is sometimes off the screen * Expanded then return to normal: zoom is not adjusted correctly If we find a way to repeat the items marked 'CAN'T REPEAT' they should go into another ticket. Maybe in DM-7068 if it is still opened. ",4 DM-7003,"Match across filters -- Make color-color diagram","Add the capability to match across filters. 1. Create color-color diagrams 2. Analyze performance metrics as a function of color.",4 DM-7004,"Add ellipticity measurement to validate_drp","Calculate the ellipticity, and the residual ellipticity (moments - PSF). Add to calculated SRD statistics. This will involve thinking about things on an image-by-image basis, which is the natural and largely SRD-specified way for considering ellipticity.",4 DM-7005,"Show the list of packages that changed from build to build linked to the git url of the latest commit","Motivated from the deviation seen from build 156 to 157 in https://squash.lsst.codes/AM1 (caused by a commit in meas_algorithms package) we can show the list of packages that changed in the current build with respect to the previous build by comparing the git commit shas and return a list of tuples with the package name and git url.",2 DM-7006,"Update squash to use bokeh 0.12.1","Bokeh 0.12 was just released and some issues are being fixed, before updating the bokeh version used in SQUASH we propose to wait for 0.12.1 release. ",1 DM-7007,"Investigate coverage of S13 databases found so far","Look at databases located at *NCSA* so far to assess if they cover the full survey. The databases to be evaluated are mentioned in [DM-6905]. According to [S13 Testing Plan|https://dev.lsstcorp.org/trac/wiki/Summer2013/ConfigAndStackTestingPlans/Instructions] the S13 DRP dataset was split into two regions with an overalp used for cross-site verification: * *NCSA*: -40< R.A. < +10 * *IN2P3*: +5 < R.A. < +55 Hence a goal of this task is to identify which previously located candidate databases and files correspond to either or both of these ranges.",4 DM-7008,"Check boost.python building with Python 3","We may want to disable boost.python in the build. There are hints that there are problems with python3.5.",1 DM-7009,"std::string construction from NULL pointer in ctrl_events","I was browsing through ctrl_events package and found couple of instances in the headers where std::string instance is constructed from NULL pointer: https://github.com/lsst/ctrl_events/blob/master/include/lsst/ctrl/events/Receiver.h#L87 https://github.com/lsst/ctrl_events/blob/master/include/lsst/ctrl/events/Transmitter.h#L81 I suspect that this code is never executed and those methods are overridden in subclasses because that construct will very likely crash when executed (std::string does not support construction from zero pointer, it will try to read from that pointer). Even if it's not executed it's better to change to return empty string or, if those two classes are never instantiated, make them abstract and make the methods pure virtual. ",1 DM-7010,"Builds should be optimised by default","By default, our builds are not optimised ({{-O0}}), which requires everyone who doesn't want to wait until the heat death of the universe to set {{SCONSFLAGS=""opt=3""}}, but other packages that are built with scons may not recognise this. This default is also contrary to the standard practise for open-source software, which is that by default builds are optimised. I will change the default optimisation level to {{opt=3}} from the current {{opt=0}}. I will also add support for {{-Og}}. This change was approved in RFC-202.",1 DM-7011,"Run DECam data through proccessCcd.py and imageDifference.py",NULL,2 DM-7012,"assign initial responsibilities in LDM-151","Assign first thoughts responsibilities to all software primitives and algorithmic components. This is my take. John will have his own take.",2 DM-7014,"Memory cache leak in firefly server","The visualization system is not update the memory accounting for the caching system.",2 DM-7015,"Analyze segmentation fault in EventAppenderTest","Analyze the bug described in DM-6462",4 DM-7016,"Big image not showing working message when the load","This is a problem with uploads, large image loads, and Atlas. When a big image is loading the user does not get feedback. The problem is the the UI is not creating the ImageViewer soon enough.",2 DM-7017,"Firefly JavaScript API documentation to support Camera team","Convert API documentation and code examples to get Camera team started with the converted Firefly FITS viewer. ",6 DM-7018,"Firefly distribution build","We need to support regular Firefly distribution builds (with bundled tomcat server), similar to the builds we did in lsst firefly repository before the conversion. This is to get Camera team started with new API.",2 DM-7019,"Setup standalone Firefly build using IPAC github","Modify the existing Firefly-Standalone build in Jenkins to use IPAC's github. Make sure github auto-releases still works.",3 DM-7021,"Update pex_exceptions to support Python 3","{{pex_exceptions}} needs to be updated to support Python 3.",1 DM-7022,"Package an experimental Firefly widget","The aim is to package an experimental Jupyter widget with limited functionality so that it can be installed like other Jupyter widgets. Only a small set of Python and Javascript code will need to be packaged -- the widget will connect to a Firefly server. The [Jupyter widget cookiecutter|https://github.com/jupyter/widget-cookiecutter] provides a template. ",4 DM-7024,"Add more features to JS9 Wrapper ","Di progress on adding extra features to JS9, including load and saving regions in the local notebook server, same with files. ",4 DM-7025,"Investigate the option to use websockets used by jupyter to explore bi-directional communication ","Di progress on understanding the possibility of using websocket locally to communicate with JS9 instances on local server. In this case we wouldn't need an external server and communication can be bi-directional. Now is only in one direction (mostly) when running Js9 locally ",3 DM-7026,"Setup up a cluster with kubernetes","Sahand progress on installing and deploying a cluster automatically with Kubernetes. After this is completed we will use a user case example of running in cluster managed by Kubernetes/Spark",6 DM-7028,"Port daf_base to Python 3","Changes necessary to get daf_base to work with Python 3.",1 DM-7029,"Image Bugs noticed in the API testing","- Image is coming with one draw layer. (I can delete this draw layer and nothing changes on the image) - When draw layer is deleted, and no more layers are present, the layers dialog should be closed. (It stays with nothing to display, you have to click x to close it.) - After selecting an area in one viewer, I select an area in another viewer, then move the mouse to the first one: 147 ImageViewerLayout.jsx:314 Uncaught TypeError: Cannot read property 'x' of null at ImageViewerLayout.jsx:314 Nothing works after that. I have to reload. - Selection appears with an offset, if the page, which contains the viewer is scrolled. (Load the attached script, press 'Start selection tracking' click to select a point, then scroll page a bit down, then click to select another point - it shows down from where it should be.) - I have 2 image viewers in separate divs. Selected line in one, then selected line in the other. The line from the first one disappeared, but its label is still there. (See attached image.) - Selection is working differently from distance. To select in another plot, I need to press selection again. I don't need to press ruler again to select new distance in another plot. - It's possible to select distance tool and then area selection. First drag would define area selection, all the following line. A click would be defining a 0 length line, even if point selection is enabled. _move to_ DM-6473 - The payload.attValue of CHANGE_PLOT_ATTRIBUTE action is using WorldPt for area and point selections (when payload.attKey is 'SELECTION' or 'ACTIVE_POINT'), but ImagePt for line selection (when payload.attKey is 'ACTIVE_DISTANCE') How can I make them all use image coordinates? ",6 DM-7030,"Update xrootd from upstream",NULL,4 DM-7031,"Assign initial responsibilities in LDM-151","Assign first thoughts on responsibilities to all software primitives and algorithmic components. This is my take. Simon will have his own take.",2 DM-7032,"Estimate resource requirements for Software Primitives","Meet with Jim & Simon. Discuss the Software Primitives section of LDM-151: clarify any ambiguities and perform an initial resource loading estimate.",3 DM-7033,"Estimate resource requirements for Software Primitives","Meet with Simon & John. Discuss the Software Primitives section of LDM-151: clarify any ambiguities and perform an initial resource loading estimate.",3 DM-7034,"Estimate resource requirements for Algorithmic Components","Meet with Jim & Simon. Discuss the Algorithmic Components section of LDM-151: clarify any ambiguities and perform an initial resource loading estimate",3 DM-7035,"Estimate resource requirements for Algorithmic Components","Meet with Simon & John. Discuss the Algorithmic Components section of LDM-151: clarify any ambiguities and perform an initial resource loading estimate",3 DM-7036,"Port pex_policy to Python 3","Changes needed to make pex_policy work on Python 3",1 DM-7037,"Check endianness in ndarray/numpy conversions","As reported on [community.lsst.org|https://community.lsst.org/t/how-to-run-the-dm-stack-on-simulated-fits-images/892/9], our {{ImageF(array)}} constructor will accept arrays with non-native endianness and interpret them as native. This probably means the array converters in ndarray aren't including byte order when checking whether a passed array's dtype matches the expected C++ type. ",2 DM-7038,"Setup JSDoc generation for the API portion of Firefly","We need generate and publish JSDoc for Firefly JavaScript API, both high and low level.",4 DM-7039,"Familiarization with Footprint redesign","Familiarize yourself with the RFC-37 driven Footprint redesign. Start thinking about ideas for how you could implement it and what the transition plan from the current Footprints might be. A great outcome would be to propose a set of stories which would tackle the new Footprint development effort. A good outcome would not be to have the stories ready to go, but to be well prepared for a discussion with [~jbosch] & [~swinbank] where we'll come up with some stories as a group.",5 DM-7040,"Stars selected by starSelector change when number of cores varies","Sogo Mineo writes in [HSC-1414|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1414]: {quote} See the following lines: meas_algorithms/HSC-4.0.0/python/lsst/meas/algorithms/objectSizeStarSelector.py:466 in ObjectSizeStarSelector.selectStars(): {code} if psfCandidate.getWidth() == 0: psfCandidate.setBorderWidth(self._borderWidth) psfCandidate.setWidth(self._kernelSize + 2*self._borderWidth) psfCandidate.setHeight(self._kernelSize + 2*self._borderWidth) {code} In reduceFrames, these lines set the width of psfCandidate to be 21 for the first time the execution reaches there. When the first CCD image has been processed, the worker process continues to process another CCD image, and the execution reaches here again. This time, psfCandidate.getWidth() is 41, because psfexPsfDeterminer has set it to be 41, and the value has been retained because the width is a static member. And so, for the second CCD image, the width of psfCandidate is not 21 but 41. Since psfCandidates are widened, stars positioned at edges of images are rejected. It results in a smaller number of PSF candidates than expected. Only CCD images that are initially given to the worker processes are processed with psfCandidate.getWidth() == 21. The other CCD images are processed with psfCandidate.getWidth() == 41. When the number of SMP cores changes, CCD images are processed with different parameters. The change in the number of PSF candidates results in different Psf, a different result of image repair, and different catalogs. {quote} The line numbers are different on the LSST side because of refactoring (objectSizeStarSelector.py:466 has moved to starSelector.py:148), but the bug is still present. The main problem appears to be that the {{PsfCandidate}} elements are {{static}}, are being set in both the star selector and the PSF determiner and one of those is conditional on what the value is. I will investigate moving the {{static}} class variables to instance variables --- the desired size appears to vary by context, so it shouldn't be a class variable.",2 DM-7044,"Additional constraints on reference band selection for multiband","Reference band selection currently depends on the configured band priority order, with exceptions made for sources with low signal-to-noise in the high priority bands. [HSC-1411|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1411] points out that some additional qualifications, such as success for major measurements (e.g., CModel and Kron), would be helpful.",3 DM-7046,"Prototype python cache for weak_ptr and weakref objects.","There is a requirement for composite datasets that components of composites be cached and shareable, and we expect to use an object cache for this. We have identified that we need to be able to cache C++ weak_ptr and python weakref in the same cache in an opaque way. This needs some prototype R&D.",2 DM-7047,"Port pex_config to Python 3","Work involved in ensuring that pex_config passes all tests on Python 3 and legacy Python.",1 DM-7048,"validate_drp is failing because it's accessing butler internals that have changed","need to change obs_decam's ingest task to use the newer class hierarchy to get the root of the butler's single repository. (longer term there should be a butler API for this or the task should get the value of root from somewhere else)",1 DM-7049,"Move patch/tract and config mapping definitions to daf_butlerUtils","Implement RFC-204 by adding new entries for all patch/tract and config mapping definitions to .yaml files in daf_butlerUtils, and removing any such entries that are identical to the common ones from .paf files in obs* packages. I think the ""common"" entry can usually be defined by consensus between any two of obs_cfht, obs_decam, and obs_subaru (and frequently all three). If there are any patch/tract or config datasets for which no two cameras agree, I think we should use obs_subaru's definitions (but I doubt there are any such cases). Entries that are not identical to the common ones should not be removed on this issue (that should make this change entirely backwards compatible), but should be documented in new per-camera issues for later standardization.",4 DM-7050,"LTD Keeper: Use Google Cloud Platform SQL","Currently LTD Keeper uses a sqlite DB. This ticket will migrate that DB to Google’s Cloud Platform’s managed SQL. This solution provides automatic backups, and provides flexibility to run multiple ltd-keeper pods. Google’s SQL makes sense since LTD Keeper is run on Google Cloud Platform.",5 DM-7051,"conda installation from the stack channel brings in astropy 1.2","If you do a conda install of lsst_sims from http://conda.lsst.codes/sims you get astropy-1.1.1. If you do the same install from http://conda.lsst.codes/stack you get astropy-1.2.1. This is a problem for the sims stack since the sncosmo package (on which we depend for our simulations of supernova light curves) is sensitive to which version of astropy you are running. Was it intentional that the two channels deliver different versions of astropy?",2 DM-7053,"Assemble a complete database with S13 DRP catalogs","Create a database populated with complete catalogs resulting from processing of the *SDSS Stripe 82* data at both NCSA and IN2P3 sites. The database has to be created at NCSA on the following database server: * *lsst-db.ncsa.illinois.edu* The database name will be: * *gapon_SDRP_Stripe82* The database will be populated with the contents of the following databases: * *NCSA* (lsst-db.ncsa.illinois.edu): {code} daues_SDRP_Stripe82_ncsa daues_SDRP_dedupe_byfilter_0 daues_SDRP_dedupe_byfilter_1 daues_SDRP_dedupe_byfilter_2 daues_SDRP_dedupe_byfilter_3 daues_SDRP_dedupe_byfilter_4 {code} * *IN2P3* (ccdb02.in2p3.fr): {code} lsst_prod_DC_2013_2 lsst_prod_dedupe_byfilter_g lsst_prod_dedupe_byfilter_i lsst_prod_dedupe_byfilter_r lsst_prod_dedupe_byfilter_u lsst_prod_dedupe_byfilter_z {code} Additional requirements: * The duplicate entries (due to the overlap in the RA range) will need to be carefully assessed and eliminated. * the referential integrity of the resulted database will need to be tested ",16 DM-7054,"Kick-off meeting","[~nlust] & [~swinbank] will meet with the SUI/T team on 2016-07-26 and discuss how we can best engage with them.",1 DM-7055,"fix miscellaneous table issues","# disable sorting when content is html # table options: auto-adjust all column width based on content # table refractoring: exposing more actions to saga. #* renamed a few actions to better reflect what it's doing. #* added TABLE_FILTER #* added document for sequence of actions where applicable. # update build script to skip buildClient when possible. # Catalog overlay should not use table id in drawing layer description. (See the attached screenshot.) It should be using MetaConst.CATALOG_OVERLAY_TYPE attribute value",4 DM-7056,"Wrap afw::table with pybind11","Following the same pattern as DM-6926, DM-6297, etc.",8 DM-7057,"Complete afw port to pybind11",NULL,10 DM-7059,"Plot sources/source density on WCS quiver plots","We should show the individual sources (size/color scaled by S/N?) that went into the jointcal fit, and/or a density map of the sources (scaled by S/N?) under the quiver plots. This will help distinguish areas with poorly constrained fits or where the TAN-SIP function diverges, from those where the new WCS really is odd.",2 DM-7060,"Plot old/new jointcal WCS vs. tangent plane","To better understand the jointcal WCS vs. the original single frame TAN-SIP, we need quiver and heatmap plots of each WCS (old and new) separately vs. a tangent_pixel or related ""non-distorted"" projection. This will let us compare the original single frame fit with jointcal's fit. This probably could be done with CameraGeom, but would be easier with the upcoming new WCS/transform system, since it may involve pulling out a new Frame.",4 DM-7061,"Plot ""real"" distortion by comparing with reference catalog","To compare the old WCS and jointcal's fit with the ""real"" distortion, we can use the matched reference catalog to plot a quiver diagram or an interpolated heat map showing how far each star is from its reference star. We may have to think about how to select objects for this plot, since centroiding errors would make it not so useful. This would probably be most useful for lsstSim, since that has an infinite-precision reference catalog.",4 DM-7062,"Support work related to PDAC effort","This issue captures emergent work to support for example DM-6905 , for which I spent some cycles locating datasets of the 2013 SDRP, staging some files off of BW tape through globus online and unpacking to /nfs/scratch, etc. This effort may not fit exactly as 'emergent middleware', but it was roughly the best fit at this time. ",3 DM-7063,"support work for testing shared stack in NFS","It was realized that the ""shared stack of lsstdev"" was not actually usable on the local condor pool due to /ssd usage. In response to this, an effort for a second shared stack on NFS was initiated in DM-6968. This issue captures the emergent work of pipeline testing to validate the new stack of that issue. ",2 DM-7065,"Extend functionality of experimental Jupyter widgets for Firefly","A package for experimental Jupyter widgets for Firefly is being developed in https://github.com/Caltech-IPAC/firefly_widgets . Using the Firefly Javascript API for Images and Tables, add some further useful functionality for demonstration purposes.",4 DM-7066,"Port pex_logging to Python 3","Work required to get pex_logging working on python 3. Will also include some package cleanups.",1 DM-7067,"Break joincal's link to upstream lsst_france repo","lsst/jointcal is still linked to the upstream repo at lsst_france. I believe all the relevant changes have been ported. It's time to break that upstream link, so that pull requests can be made in a more obvious fashion.",1 DM-7068,"Firefly API bugs 2","*Issues* Gator: * Missing feature: before migration, in expanded mode, the toolbar had an 'added image' button which was bringing an image search panel to add images to the current view. * The Gator Multi-object search seems having problem with the coverage image. Atlas: * if marker/footprint overlay is clicked, that doesn't activate the image viewer and doesn't update the layer dialog either. Large drawing layers block viewer from becoming active, WFIRST footprint or WFC3/IR cause the problem. * -in expand mode and zoom 'fill the visible space' clicked, the magnifier image doesn't show anything from the coverage- - not a bug, magnifier is disable when zoom level is above 8x API: * In API, it is not possible to drag dialogs (ex. Drawing Layers). All Firefly (found by Tatiana): * The highlight should of catalog or coverage overlay should just change if you are close to the point. For now I am setting it to 20 pixels * -Catalog overlay should not use table id in drawing layer description. (See the attached screenshot.) It should be using MetaConst.CATALOG_OVERLAY_TYPE attribute value.- _Moved to_ DM-7055 * After using distance tool in one plot, then the other, clicking again in the first plot does not make it active. (You have to click on the border of the plot to make it active, clicking inside does not help.) This seems to cause strange behavior, when selections do not work as expected. For example: select ruler, select some lines alternating first and second plot. Unselect ruler, select area icon. Selecting in the first plot will still show distance. I had to delete distance tool drawing layer or click on the border for things to start showing area selection. In general, there is some confusion with active plot, when I have two viewers. Should a plot become active as soon as mouse enters is? Otherwise in readout, compass thumbnail will still show active plot, while readout thumbnail could use another plot. _note from trey: this is the same problem as the marker/footprint in atlas described aboved_",4 DM-7069,"Port daf_persistence to Python 3","Work relating to getting daf_persistence to run on python 3. Includes some code modernization.",1 DM-7070,"Move consts from top of Associations.cc into JointcalConfig","There are three values at the top of Associations.cc under a TODO comment that should be lifted up into JointcalConfig so they can be configured at runtime. It would be good to try to add tests to check different values for them (and possibly just remove usnoMatchCut).",1 DM-7071,"Fix Django admin interface ","Django admin interface is useful to edit db entries in SQUASH if needed, e.g decam measurements that were incidentally pushed to the dashboard during X16. A bug was found using the admin interface in development mode, due to a bad field returned by the Jobs model. This ticket is to capture the fix for this bug, this new git ref will be deployed for better control of data in SQUASH database. ",1 DM-7072,"visit DRP team, June 2016","Travel to Princeton June 13-17 and meet with the DRP team; work and learn about L2 processing; discuss the workflow requirements and use cases. ",8 DM-7073,"Install ESXi on lsst-dm-mac.lsst.org","Install and configure ESXi on the Mac Pro server.",7 DM-7074,"Install Mac OS X Mountain Lion on ESXi","Install, configure and snapshot Mac OS X Mountain Lion. Unfortunately this is required to install any other Mac OS VM on ESXi.",1 DM-7075,"Install Mac OS X Yosemite on ESXi","Install, configure and snapshot Mac OS X Yosemite. This requires configuring the vmx file then installing Mac OS X Mountain Lion and upgrading.",2 DM-7076,"Install Mac OS X El Capitan on ESXi","Install, configure and snapshot Mac OS X El Capitan. This requires configuring the vmx file then installing Mac OS X Mountain Lion and upgrading.",2 DM-7077,"Install MacOS Sierra on ESXi","Install, configure and snapshot MacOS Sierra. This requires configuring the vmx file then installing Mac OS X Mountain Lion and upgrading.",1 DM-7078,"Firewall and SSH configuration on ESXi","Figure out and configure the firewall, ssh server and ssh client for ESXi. This isn't especially well documented since it's part of VMWare vSphere. This part specifically was time consuming since most users by vSphere.",2 DM-7079,"Upgrade panopticon to 5.0.0-alpha4","This upgrade requires moving from Topbeat to Metricbeat which requires some minor rework and upgrading the entire system at once.",2 DM-7080,"Doxygen isn't updating","The current build of our Doxygen documentation, as displayed at https://lsst-web.ncsa.illinois.edu/doxygen/x_masterDoxyDoc/index.html, is labelled ""Generated on Mon Jun 27 2016 03:52:22 for LSSTApplications"". At time of writing, that's more than a month ago. Important additions to the documentations made during the last month are missing. ",1 DM-7081,Airflow,"Review [Airflow|https://github.com/apache/incubator-airflow] workflow management system against criteria defined in the epic.",6 DM-7082,"deploy django admin interface fix","Test and deploy django admin fix from DM-7071.",1 DM-7083,"Install MySQL and PostgreSQL servers on ccqserv124","Time to run my incomplete L! prototype on real hardware, for that I need MySQL and PostgreSQL servers on a dedicated machine in in2p3 cluster (ccqserv124). Probably start with installing what comes with the system repos before trying latest and greatest stuff. Both servers need to be configured to allow me create databases/tables, and I only need to enable connections from localhost. ",1 DM-7084,"Astropy views not available on Catalog subclasses","Somehow the {{asAstropy}} isn't being inherited by {{BaseCatalog}} subclasses in Python; it's probably getting messed up by the fact that {{Catalog}} is a template and this is added at the Swig level.",1 DM-7088,"Image select panel not yet working correctly with coverage","The image select panel needs to be able to modify the coverage image.",4 DM-7090,"IrsaViewer catalog panel, labels and input fields moved as you type","Catalog search panel in IrsaViewer, the target panel label, feedback, and input box are jumping as input is being typed. Their position should be fixed.",2 DM-7091,"F16 Qserv Release Mgmt","Developer work to support the monthly and end-of-cycle qserv releases. Includes compiling release notes, updating package dependencies, updating installation docs, minor fixes in support of new compilers, etc.",20 DM-7094,"Develop Sphinx configuration for Pipelines Documentation, including MVP HTML/CSS Template","This ticket will kick-off a pilot implementation of pipelines documentation in Sphinx. Specific goals are 1. Develop template for sphinx-ready doc/ directories in packages (based on SQR-006) 2. Setup a MVP sphinx template that works well with numpydoc and astropy automodsumm. Simply porting astropy’s sphinx template would be pragmatic. 3. documenteer-driven configuration for sphinx. These will be MVPs, and iterated upon in later tickets that implement sphinx API docs for stack packages.",1 DM-7103,"Run DAX containers at NCSA","This is an initial step to manually launch the containerized DAX services on the new PDAC cluster. This is meant to expose container configuration, account setup, privilege, logging, debugging, etc. issues.",4 DM-7104,"support PDAC Qserv deploy","Support John in adapting scripts and methodology as necessary to support qserv deploy on the PDAC cluster at NCSA, as is currently done at IN2P3. ",8 DM-7105,"Qserv 2016_08 release",NULL,1 DM-7106,"PDAC Qserv Deploy","Configure cluster and adapt scripts and methodology as necessary to support qserv deploy on the PDAC cluster at NCSA, as is currently done at IN2P3. ",13 DM-7107,"Deliver revised slides for Joint Status Review","Deliver a modified version of the slides from the July 2016 Joint Directors (sic) Review, plus service any other requests from Project Management.",6 DM-7108,"Provide updated F16 DRP plan for PMCS ingest","Only the first three months of F16 were concretely resource loaded and ingested into PMCS at the start of the cycle. A provisional plan was loaded for the remaining three months. Check, refine and update than plan as necessary.",4