| 1 | issuekey | title | description | storypoint |
|---|
| 2 | DM-4 | Transition git repositories to Stash | Transition gitolite-managed repositories to Atlassian Stash. | 10 |
| 3 | DM-8 | Finalize DM mission statement | The proposed mission statement for the LSST Software Stack development: {quote} Enabling LSST science by creating a well documented, state-of-the-art, high-performance, scalable, multi-camera, open source, O/IR survey data processing and analysis system {quote} with the following rationale: * well documented -- as otherwise it will be impossible to maintain or be usable for Level 3 * state-of-the-art -- because the quality of the instrument and needs of the science to be systematic limited require us to develop new algorithms an break new ground in a number of areas * high-performance -- because of the massive amount of data we need to process for a reasonable budget * scalable -- because we need it to run from between a single core (for developers, or Level 3 users) to tens of thousands of cores (for Data Release production) * multi-camera -- because we need to process precursor data for development purposes, and because our as-delivered camera won't be ideal. * open source -- because we want the community and future surveys to benefit from our efforts. | 2 |
| 4 | DM-9 | Open up LSST software mailing lists | We all benefit from making LSST software development as open as possible and conducive to outside volunteer contributions (*). One way to increase community involvement is to open up our development mailing lists to the public, analogous to the way other open source projects do. For example, we could have: * software-devel@lsstcorp.org: the development mailing list, equivalent to current lsst-data * software-users@lsstcorp.org: the users mailing list, equivalent to current lsst-dm-stack-users mailing list (but it could possibly be replaced by StackOverflow/Confluence Questions) * lsst-dm@lsstcorp.org: internal, DM-staff only mailing list, for the *rare* discussions/notices that should go out to staff only. (*) Though we don't rely on them for meeting the project specs (legally required disclaimer :) ). | 1 |
| 5 | DM-10 | Transition to Confluence Questions | Open the Confluence Questions site, for interaction with the community (and to enable community self-help). | 10 |
| 6 | DM-17 | Add derivatives-based optimizer to meas_multifit | See https://dev.lsstcorp.org/trac/ticket/3146 Story points estimate is for remaining work only (just the code review, which is still substantial). | 10 |
| 7 | DM-20 | Release EUPS 1.3.0 | Release EUPS 1.3.0 in RHL's github repository. | 1 |
| 8 | DM-21 | Determine final URL/location for W'14 stack | Need to know where W'14 stack files are going to be housed. | 1 |
| 9 | DM-22 | Update newinstall.sh to check for existence of git and python on users' machines | NULL | 1 |
| 10 | DM-23 | Confirm stack builds on OS X 10.8 | NULL | 1 |
| 11 | DM-25 | Build Winter'14 release | Run lsst-build scripts and 'eups distrib create' to build the Winter'14 release. | 1 |
| 12 | DM-26 | Update installation instructions | Update Confluence instructions on how to install the Winter'14 stack. The instructions are at: https://confluence.lsstcorp.org/display/LSWUG/LSST+Software+User+Guide | 1 |
| 13 | DM-28 | Modify gitolite permissions to allow issue/DM-NNNN branches | Use issue/DM-NNNN branches for issues tracked in JIRA, to differentiate them from tickets/NNNN branches that are still in Trac. | 1 |
| 14 | DM-52 | Qserv configuration - detailed design | Detailed design covering how all Qserv components will be configured for runtime. | 3 |
| 15 | DM-53 | Node configuration and bootstrapping - detailed design | Design covering how all Qserv components will be configured for runtime. Design how new Qserv nodes will be bootstrapped when we add them to the cluster, and how already added nodes will get updated after they were offline (crashed, turned off for maintenance etc) | 6 |
| 16 | DM-55 | Node bootstrapping - 1st prototype | NULL | 15 |
| 17 | DM-56 | Zookeeper-based CSS (v1) | NULL | 40 |
| 18 | DM-66 | switch back to throwing exceptions in css/Facede.cc | NULL | 1 |
| 19 | DM-70 | Rewrite xrootd-facing code | This task is related to DMTF-16570-09. It involves plugin in the new XRootD client (which has different interfaces, it is all async etc). We expect to also do a major cleanup on the code that is talking to XRootD while plugging in the new XRootD. | 30 |
| 20 | DM-71 | Data Distribution Design v1 | Need to come up with detailed design covering how we will deal with data distribution: managing multiple replicas, recovering from faults, adding new nodes to the cluster, registering new data from L2 ingest and user data (L3). | 8 |
| 21 | DM-75 | Modify format of version numbers | The versions auto-generated by the new EUPS+buildbot look like this: {code} ========== $ eups list .... pipe_tasks 7.3.2.0_5_g455c355d0f+070b2c1b35 b61 pyfits 3.1.2+9ef17db9b7 b61 b60 python 0.0.1 b61 b60 scisql 0.3+2b5a2f1b52 b61 b60 scons 2.1.0+2b5a2f1b52 b61 b60 sconsUtils 6.2.0.0_11_gf38997df3e+759c3944a1 b61 sconsUtils 6.2.0.0_19_g755151c0a5+759c3944a1 b60 shapelet 7.3.1.0_1_g9331ee763c+0c72f294dd b61 shapelet 7.3.1.0_1_g9331ee763c+e56ee84a68 b60 skymap 7.3.1.0_1_g64b750c066+db36490146 b60 skymap 7.3.1.0_1_ga6cd540cd3+493a438aa2 b61 skypix 6.1.0.0_1_g1157bf09ae+65137c93cd b61 skypix 6.1.0.0_1_gea33592463+6039c04989 b60 .... ========== {code} where the part before the plus sign is the output of git describe (slightly mangled), and the part after is the SHA1 of the sorted names+sha1s of the dependencies. While this has the benefit that any two causally disconnected buildbots with the same inputs will build the same versions, many people have complained that they're plain ugly. So here's an alternative proposal: * If a tag exist on a commit, use <tag> as the version. * If there's no tag, use branchname-gSHA1ABBREV, where any illegal characters in branchname get turned into dots * If the package has dependencies, and a build of this package with different dependencies already exists, append a +N to the end. Keep the mapping of +N -> (dependency name, sha1s) in a special git repository. Given the source code and this git repo, two causally disconnected buildbots will again generate the same set of versions. Example versions: * 7.10.2.1 * 7.10.2.1+5 * master-gdeadbeef * feature.dm-1234-gdeadbeef * feature.dm-1234-gdeadbeef+3 | 1 |
| 22 | DM-78 | Save a git-branch when a forced push is detected | Create a gitolite hook that will save a branch when a forced push is detected. E.g., if we have a ticket: 'tickets/DM-AAAA' and someone rebases it and pushes with '--force' before applying the update --- then the hook will branch off the old state into (say): backups/tickets/DM-AAAA/NNNN where NNNN is a monotonically increasing number (per branch). | 1 |
| 23 | DM-83 | from __future__ import division breaks division of Extent* | If one does: {{from __future__ import division}} the division operator on Extent types raises an exception. How to repeat: I've tried this with v7_3 and master: {code:py} from __future__ import division import lsst.afw.geom as afwGeom npt = afwGeom.Extent2I(10,10)/2 {code} an exception is raised. Removing the first line succeeds as expected. | 4 |
| 24 | DM-85 | Measurement - Aperture Corrections | The current implementation of aperture corrections in meas_algorithms' CorrectFluxes class is broken, and should not be replicated in meas_base: - it only works when the PSF model is correct - it doesn't work when the aperture to correct to is larger than the PSF model image size - it doesn't propagate the uncertainty in the aperture correction We probably need to do this by estimating the aperture correction and its errors on single frames, using the PSF stars (not the PSF models), then attaching that information to the Psf object to be retrieved *and coadded* by CoaddPsf. JK: In PMCS this would be 10% Bosch J 50% Krughoff S and 40% Owen R Breakdown: jbosch 10%; krughoff 50%; rowen 40% | 47 |
| 25 | DM-90 | Publish Winter 2014 binaries | NULL | 3 |
| 26 | DM-92 | tests/testPsfDetermination.py has a broken test | In meas_algorithms tests/testPsfDetermination.py has a test testRejectBlends which does not operate as expected. When it calls pcaPsfDeterminer it results in no usable psf candidates BEFORE blends are rejected. Formerly this resulted in a numpy array named "sizes" containing one uninitialized value, which might raise an unexpected exception or raise the desired exception, depending on whether that value was negative or positive. On tickets/DM-3117 I pushed a fix for the bug that caused the invalid "sizes" array, but the unit test is now reliably broken because no viable psf candidates raises the wrong exception and does not test blend rejection in any case. So on this same ticket I have commented out the bad test for now. | 2 |
| 27 | DM-94 | Configure transition screens for DM agile workflow | Whenever an issue is transitioned on JIRA Agile board to 'Ready for Review', a screen should pop up to ask for a reviewer. Whenever it's moved out of that state, another screen should ask for a new assignee. | 1 |
| 28 | DM-95 | Make lsst-build reuse buildIDs if nothing's changed | All built packages are EUPS-tagged with build IDs (the bNNN EUPS tags). Without this change, new EUPS tags are declared even when nothing changed since the previous build (and EUPS' tags code doesn't scale well at this time). This will be implemented by comparing the newly built manifest against ones stored in versiondb, and reusing the build IDs if a matching one is found. | 1 |
| 29 | DM-96 | Write unit tests for lsst-build | Unit tests need to be written for lsst-build; they should've been written together with the code, but due to Winter'14 release fire drill they had to be postponed. | 10 |
| 30 | DM-98 | clean up isr utility code | There is some commented code in isr.py. This should be removed or updated so that it works. | 2 |
| 31 | DM-148 | Improve naming of getters in AmpInfoTable | The names of the methods to get values from a record on AmpInfoCatalog are potentially confusing. This is because the convention is to call the getters get[attributename]. We could change the method names in the AmpInfoCatalog, or add methods in the SWIG wrapper. | 1 |
| 32 | DM-177 | Box2I(bbox.getMin(), bbox.getMax()) fails for an empty bbox | Empty Box2I cannot be round tripped: {code:py} from lsst.afw.geom import Box2I b1 = Box2I() b2 = Box2I(b1.getMin(), b1.getMax()) assert b2.isEmpty() {code} It is confusing and surprising that this round tripping fails. It is also a trap for the unwary because saving min and max is the logical way to store boxes in afw tables. Records can contain points but not extents and so it saves casting back and forth and simplifies and clarifies the code to save max instead of extent. Thus that is the path most users will take, and the problem can be a time bomb: it could be quite some time before somebody tries to store an empty box and finds that it does not get retrieved correctly. | 1 |
| 33 | DM-195 | log4cxx-based logging prototype - v2 | This is continuation of DMTF-16570-16. Initial work (v1) was done in branch u/bchick/protolog. V2 will include comments sent by K-T and issues discussed at the Qserv meeting March 13 + Free functions vs. a log object need to be discussed more. In particular, when metadata key/value pairs need to be attached, an object might make more sense. Avoiding the getLogger() call when no logging is needed (due to threshold) can be significant. + It's a security breach to use vsprintf() with any user-provided arguments. Use vsnprintf() instead so that you can check for overflow. (Or use stringstream or boost::format.) + In my prototype, I used a combination of a set of static log4cxx::LevelPtr variables with isEnabledFor(level) and a set of cpp macros to avoid the switch. + The return value from getLogger() shouldn't need to be cast. ---- + play with hierarchical names + don't execute code for formatting if debug level is off + use shorter threadId + experiment with defining special python handler, intercept and redirect to our log4cxx-based logging | 15 |
| 34 | DM-197 | Replace PositionFunctor with some flavor of XYTransform | afw has a special functor PositionFunctor that acts like an XYTransform. Unless PositionFunctor does not need to be invertible, it makes sense to merge these, likely by replacing PositionFunctor with the transform from afw::image::XYTransformFromWcsPair (as suggested by Jim Bosch on Trac ticket #2214). | 1 |
| 35 | DM-198 | Rework JOIN support, including Ref*Match tables | Add support to the Ref*Match tables. The relevant code in Qserv core (supporting joins) has already been written. This task is related to DMTF-1640-20 | 16 |
| 36 | DM-199 | Develop new master-worker result system | Reimplement how results are returned from worker to the czar. Currently it relies on mysqldump, which is fairly inefficient. This is related to DMTF-1650-045 | 20 |
| 37 | DM-201 | Qserv: unit testing (controller module) | Design and build toy prototype of a test framework for testing controller module. This might require a mock framework, as we want to be able to test things in isolation, without testing everything around the controller module at the same time. This is related to DMTF-16570-20. | 10 |
| 38 | DM-202 | Qserv: unit testing (query execution) | Design and build toy prototype of a test framework for testing query execution module. This might require a mock framework, as we want to be able to test things in isolation, without testing everything around the query execution module at the same time. This is related to DMTF-16570-21. | 8 |
| 39 | DM-203 | Prepare for setting up new cluster at IN2P3 for continuous integration/testing | Once the hardware is available, setup the environment where we could easily run integration testing of different Qserv releases, including testing/comparing performance. Integrate changes implemented in DM-1078 Add install script that exposes individual steps and allows modifications to the config file: newinstall, qserv-configure --prepare, then edit config file (or copy from somewhere), qserv-configure | 4 |
| 40 | DM-207 | Migrate Qserv czar code to the new logging system | This includes switching Qserv to the new logging. Fine-tuning (what messages are printed deciding on error level) is covered in DM-685. | 10 |
| 41 | DM-208 | catch exceptions from CSS | NULL | 1 |
| 42 | DM-210 | S15 Data ingest | Rework existing scripts used to load data into plain mysql. This involves mostly simplifying it, and pushing some functionality like converting types outside of the loading scripts. JK: Refer to loading spreadsheet for PMCS assignments | 10 |
| 43 | DM-211 | Revise design for Qserv front-end rearchitect | Revisit the architecture. This includes proxy, down to XRootD client (mysqld, python, zookeeper). Capture all findings in new stories, add these stories to DM-1707 | 10 |
| 44 | DM-212 | Migrate away from using env variables in Qserv | Qserv is currently relying on many env variables. We should migrated away from that to the extend possible. | 10 |
| 45 | DM-213 | Setup multi-node testbed | It'd be useful to test Qserv using Winter2014 or Summer2014 data set on a multi-node cluster, just to exercise all pieces of the software and double check we are not missing anything. | 5 |
| 46 | DM-214 | W15 C++ geometry | Port the geometry related code used by Qserv (it is currently written in python) to C++, and switch Qserv to the C++ based version. JK: Refer to loading spreadsheet for PMCS assignments | 47 |
| 47 | DM-215 | Implement C++ geometry primitives for Qserv | NULL | 10 |
| 48 | DM-216 | Switch Qserv to C++ geometry primitives | NULL | 10 |
| 49 | DM-218 | Experiment with no-subchunking based approach | NULL | 10 |
| 50 | DM-219 | Qserv worker scheduler – code cleanup | The qserv worker scheduler code is a bit ugly. The actual composable scheduler classes (FifoScheduler, ScanScheduler, GroupScheduler, BlendScheduler) might be pretty clean, but the interactions with the rest (wdb, wcontrol) may be harder to understand. There should be a small amount of low-hanging fruit of code to clean up, but to make things more sensible and understandable may require some new abstractions and shuffling of logic to new/different classes. Since this issue was opened, some refactoring work has been done as part of the new xrdssi port, so the organization may be somewhat cleaner now. Still, it's worth it to take a fresh look to evaluate the design/interactions to see how much can/should be reorganized. | 6 |
| 51 | DM-221 | Implement new (async) XrootD client | (oops, earlier description was meant for DM-878. Sorry for the confusion. -danielw) (I also made a mess with the assignees. I'm sorry. -danielw) | 30 |
| 52 | DM-222 | Reference Test Server using new XRootD | NULL | 5 |
| 53 | DM-224 | Switch to MariaDB | We should switch Qserv to the MariaDB Foundation based MySQL. | 3 |
| 54 | DM-228 | Setup dev test environment | Setup whole Qserv environment, including installing data set, and validate it by running some simple queries. Suggest changes/improvements as appropriate. | 8 |
| 55 | DM-231 | Refurbish existing configuration | Refurbish existing configuration scripts to make them work with the new packaging/build system. | 10 |
| 56 | DM-240 | meas_base plugins for CModel magnitudes | Create meas_base Plugins for single-frame and forced measurement that uses the model-fitting primitives in meas_multifit to implement SDSS-style CModel magnitudes, in which we fit an exp and dev model separately and then fit the linear combination with ellipse parameters held fixed. An old-style plugin has already been implemented on the HSC fork, and should be used as a guide; this issue involves adapting that implementation to meas_base and potentially cleaning it up a bit. Note that the HSC implementation cannot be transferred directly to the LSST side because the meas_algorithms APIs are slightly different on the two forks. | 6 |
| 57 | DM-241 | refactor forced tasks into two tasks | After looking at it a bit more, I think we should refactor the current meas_base forced photometry task to separate the CmdLineTask from the Measurement task. This will allow the forced measurement task to share a common base class with SingleFrameMeasurementTask (allowing us to move the callPlugin free functions into that base class), and give us better parallels with existing tasks: - ProcesImageForcedTask (my proposed name for the base command-line task) will be more similar to ProcessImageTask. We'll also have ProcessForcedCcdTask and ProcessForcedCoaddTask. - ForcedMeasurementTask will be more similar to SingleFrameMeasurementTask. In short, I think this will both clean up the ugliness in callPlugin and make the whole hierarchy easier for newcomers to understand. | 6 |
| 58 | DM-242 | switch from '.' to '_' in afw::table fields | We've been mapping '.' to '_' in afw::table I/O, which unnecessarily complicates lots of things. We'd like to switch to using '_' in the field names themselves, which requires ending this mapping in I/O, but we need to be backwards compatible. So we'll add a version to the FITS headers, and continue the mapping if the version is not present or is less than some value. Until we do this, the new field names being used in meas_base won't round-trip. | 3 |
| 59 | DM-245 | Implement HSC camera in new camera framework | The HSC camera needs to be put in the new camera geometry. I will implement is like we did lsstSim and sdss. That is I will generate a repository with the camera config and ampInfo FITS files that can be unpersisted by the butler. These changes should only require modifying obs_subaru. My plan is to just use the policy file to populate the new geometry, but if there is more up to date information, I'll happily use that. | 10 |
| 60 | DM-246 | Investigate compensation for Dcr | This is a continuation of the W14 image differencing work. We have characterized the negative effects of Dcr on difference images, and now need to start working on compensation for these effects. This work will also touch on the Wcs and Psf classes, which are probable consumers of this information. Scope includes designing and implementing a class to describe the astrometric effects of Dcr, with consideration as to the other classes (Psf, Wcs) and tasks (ImageDifferencingTask) that may use it | 60 |
| 61 | DM-247 | Design of Dcr Class | Describes the design process for implementation of a class to model the effects of Dcr. Includes design itself, and the design review process. | 20 |
| 62 | DM-248 | Implementation of Dcr Class | Core implementation of the class that represents the effects of Dcr. This only includes the initial implementation. We should realistically expect that this class will evolve as it encounters more use cases, no matter how thorough the design process is. | 20 |
| 63 | DM-259 | Test and migrate to swig 3.0 | -------- Original Message -------- Subject: [LSST-data] Swig 3.0 is out (with C++11 support) Date: Mon, 17 Mar 2014 08:26:05 -0400 From: Robert Lupton the Good <rhl@astro.princeton.edu> To: LSST Data <lsst-data@lsstcorp.org> I tried a pre-release on os/x 10.7.5 and it failed some tests, but I haven't tried this version. I had some discussion about this with William, but haven't had time to follow through. R > Date: Sun, 16 Mar 2014 22:44:42 +0000 > From: William S Fulton <wsf@fultondesigns.co.uk> > > *** ANNOUNCE: SWIG 3.0.0 (16 Mar 2014) *** > > http://www.swig.org > > We're pleased to announce SWIG-3.0.0, the latest SWIG release. > > What is SWIG? > ============= > > SWIG is a software development tool that reads C/C++ header files and > generates the wrapper code needed to make C and C++ code accessible > from other programming languages including Perl, Python, Tcl, Ruby, > PHP, C#, Go, Java, Lua, Scheme (Guile, MzScheme, CHICKEN), D, Ocaml, > Pike, Modula-3, Octave, R, Common Lisp (CLISP, Allegro CL, CFFI, UFFI). > SWIG can also export its parse tree in the form of XML and Lisp > s-expressions. Major applications of SWIG include generation of > scripting language extension modules, rapid prototyping, testing, > and user interface development for large C/C++ systems. > > Availability > ============ > The release is available for download on Sourceforge at > > http://prdownloads.sourceforge.net/swig/swig-3.0.0.tar.gz > > A Windows version is also available at > > http://prdownloads.sourceforge.net/swig/swigwin-3.0.0.zip > > Please report problems with this release to the swig-devel mailing list, > details at http://www.swig.org/mail.html. > > Release Notes > ============= > SWIG-3.0.0 summary: > - This is a major new release focusing primarily on C++ improvements. > - C++11 support added. Please see documentation for details of supported > features: http://www.swig.org/Doc3.0/CPlusPlus11.html > - Nested class support added. This has been taken full advantage of in > Java and C#. Other languages can use the nested classes, but require > further work for a more natural integration into the target language. > We urge folk knowledgeable in the other target languages to step > forward and help with this effort. > - Lua: improved metatables and support for %nspace. > - Go 1.3 support added. > - Python import improvements including relative imports. > - Python 3.3 support completed. > - Perl director support added. > - C# .NET 2 support is now the minimum. Generated using statements are > replaced by fully qualified names. > - Bug fixes and improvements to the following languages: > C#, Go, Guile, Java, Lua, Perl, PHP, Python, Octave, R, Ruby, Tcl > - Various other bug fixes and improvements affecting all languages. > - Note that this release contains some backwards incompatible changes > in some languages. > - Full detailed release notes are in the changes file. | 2 |
| 64 | DM-260 | Board workflow modifications | On DM Software Development board, I propose we: * Change "Ready to Merge" to "Review Complete". Per K-T: {quote} That's what I thought "Review Complete" would be -- most of the work is done, but some fixups are needed before merging, and a re-review is not necessary. {quote} * Remove "Ready for Review". Instead, the developer should just drag the issue to "In Review" and assign it to a reviewer. If/when we re-instate the "review master", we may re-introduce "Ready for Review" | 1 |
| 65 | DM-271 | Setup the new Buildbot CI system | Setup the Buildbot 0.8.8 testbed for the DM environment. This includes: (1) setting up slaves on the set of common OS on which the DM stack runs; (2) creating a new continuous integration slave using the new eupsPkg-based build and distribution support, Definition of done: * Every git change of master should trigger a build of master * If a build failed, an e-mail will be sent to lsst-data (if the build succeeds, nothing happens) * Failures due to internal buildbot issues (e.g., config problems, transient system availability issues, timeouts, etc.) should go to the buildbot owner. * Allow user-triggered builds via web page (with specified refs to be built), with a common user (until we get LSSTC LDAP directory hooked up). It's understood that locking may not be fully implemented/fleshed out in this story. * It should be possible for a user on lsst-dev to easily setup the stack for either a failed or a successful build. | 100 |
| 66 | DM-272 | Move TCT-relevant twiki documentation to Confluence | Congregate all the trac TCT-relevant documents (standards, policies, guidelines, meeting history) onto Confluence. | 2 |
| 67 | DM-273 | Develop and then create the organizational structure for DM Confluence space | Before we start populating the DM Confluence space with active pages, we should define an overall organizational structure/taxonomy. | 1 |
| 68 | DM-274 | Adding support for numpy scalar array types as arguments to SWIG wrapped methods | Building against the anaconda on lsst-dev causes construction of Point2I (and other point types) with numpy dtypes to fail with the standard exception: {code} NotImplementedError: Wrong number or type of arguments for overloaded function {code} I did not know that was not allowed. If I am doing it others are as well. I think this should be addressed in some way before W14 release. How to repeat on lsst-dev: {code:sh} setenv EUPS_DIR ~lsstsw/stack/ source ~lsstsw/eups/bin/setups.csh setup afw setup anaconda echo "import lsst.afw.geom as ag\nimport numpy\nprint ag.Point2I(5,5)\nval = numpy.int64(5)\nprint ag.Point2I(val,val)" > test.py python test.py {code} Using the old 7_3 stack you can do a similar test: {code:sh} source /lsst/DC3/stacks/default/loadLSST.csh setup -t v7_3 afw echo "import lsst.afw.geom as ag\nimport numpy\nprint ag.Point2I(5,5)\nval = numpy.int64(5)\nprint ag.Point2I(val,val)" > test.py python test.py {code} | 3 |
| 69 | DM-278 | Improve handling errors occuring in AsyncQueryManager | AsyncQueryManager is initialized based on configuration file, if the configuration is invalid, an exception should be thrown (eg in _readConfig()) and gracefully handled upstream. | 1 |
| 70 | DM-280 | clean up multiple aperture photometry code | I've been doing some minor work on the HSC-side ApertureFlux algorithm, and I wanted to record some concerns here (from both me and RHL) that should be addressed in the new meas_base version: - We should consider merging ApertureFlux and EllipticalApertureFlux into the same algorithm (with a config field to choose whether to use elliptical apertures). We could still register it twice, with a different default config value, and this should eliminate a lot of code duplication. We could also consider having them inherit from a common base class (instead of having EllipticalApertureFlux inherit from ApertureFlux, as is done now). - We should test that the threshold at which we switch from Sinc to naive apertures is obeyed exactly. - We should create a flag for the failure mode in which an aperture cannot be measured because we go off the edge of the image, and test that it appears at the right point. If possible, we should set this flag and measure what area we can within that aperture, instead of just bailing out. - (Somewhat off-topic) We should consider having utility functions on SourceSlotConfig to set all slots to None for use in unit tests. | 5 |
| 71 | DM-289 | More helpful location information for errors in duplicator/partitioner input | Currently, if the Qserv duplicator or partitioner encounters erroneous input (such as a formatting error, or missing columns), the error message it outputs does not include a mapping back to the location of the error, making it very hard/annoying indeed to fix that erroneous input. Fabrice would (quite reasonably) like to see a filename and line number in the error message. Unfortunately, producing a line number is complicated by the following facts: - multiple threads may be reading sub-sections of the same input file in parallel - sub-sections are not guaranteed to be read in order - lines can have varying length In other words, the code reading/parsing the input has no idea which line it is working on, and making that information available would involve either deferring error reports or extra synchronization (performance loss). What we can easily (and should) do is to arrange for processing code to know the file and byte offset of the input text being processed; error messages should include both pieces of information. | 2 |
| 72 | DM-290 | Eliminate dependence of query analysis on parser and antlr | I would like to write and compile query analyzer code completely independently of the parser and ANTLR (transitively). This doesn't seem to work right now. This is not currently possible. This might take any where from a day to a week. (I'm not sure if we can finish anything in half a day, if you include the testing, review, feedback, and revision process, but perhaps unit testing will make that faster). Updates to follow after the scope is estimated. Dependencies to be broken: query --> parser, antlr (due to predicate depending on antlr nodes) qana --> parser, antlr | 8 |
| 73 | DM-291 | Investigate new XrdSsi interface | NULL | 10 |
| 74 | DM-295 | xrootd initialization should abort if mysql connection fails | Currently xrootd will happily start even if it can't connect to mysql, it will only print a message: {code} Configration invalid: Unable to connect to MySQL with config: {code} This can be easily overlooked, plus, it is a fatal error and xrootd initialization should be aborted. while working on this, I propose to also improve validateMysql(). At the moment it connects to mysql and database in one call. If connection is fine, but the database does not exist, it will fail without telling user why it failed. This can be confusing. It'd better to connect to mysql without connecting to database, then do "select_db", and if that fails, inform the user that the database does not exist. (transferred from trac ticket 3165) | 1 |
| 75 | DM-296 | fix namespaces in all Qserv core modules | This was suggested by the code review for ticket 1945 (https://dev.lsstcorp.org/trac/wiki/SAT/CodeReviews/1945), pasted below: common/src/*: While it's not required by the coding standards, I'm a big proponent of using namespace scopes in .cc files, which usually save you from needing namespace aliases and will certainly save you from having prefix every declaration with qserv::. At some point I'd recommend changing the header file extension from .hh to .h to match the rest of the LSST DM code, unless it's a big backwards compatibility issue. (transferred from trac ticket 2528) | 5 |
| 76 | DM-297 | Qserv should check for loaded spatial UDFs | Qserv should have a way of checking for the existence of spatial UDFs loaded in the worker MySQL instances. Obviously, the worker must perform the physical check. However, the worker has no knowledge that spatial UDFs even exist, since the master is responsible for translating spatial spec into UDF call. At the moment, the preferred way of checking would be some sort of administrative command that runs on all nodes. An alternative would be some sanity check that is run on a worker before an admin starts a worker. Or the master could devise a MySQL query to check for things and dispatch to all chunks (although this would not get full coverage when replica exist, while being redundant while workers host more than 1 chunk). (transferred from trac ticket 1959) | 2 |
| 77 | DM-298 | restarting mysqld breaks qserv | Restaring mysqld results in unusable qserv (even if the restart happens when qserv is completely idle). The error message is: ERROR 2013 (HY000): Lost connection to MySQL server during query This happens most likely because qserv caches the connection, which becomes invalid when server is restarted. I am guessing the same will happen when there is a long period of inactivity (the connection times out). (transferred from trac 2853) | 1 |
| 78 | DM-300 | Centralize hardcoded constants | Some values in qserv need to become constant(e.g. chunkId column names, dirs, filenames). Some of these are configurable, others are hardcoded in non-obvious places in the code. When multiple places need this value, they really need to agree, and unfortunately, qserv doesn't have a well-known place for these constants yet. Any (constant) value that is needed by different parts of the code needs to be managed in a way that is reasonably obvious to unfamiliar programmers. List of values: * chunkId, subChunkId column names (master...indexing.py, app.py) * environment variable names * + others. This should actually be fairly simple to implement, once the right (?) design is conceived and worked-out. (transferred from trac ticket #2405) | 6 |
| 79 | DM-309 | Jira for Qserv | Jira setup for Qserv, includes things like adding new tasks, transferring tasks from trac, epic/story/task division, assigning story points, setting scrum board, just learning things and more... | 8 |
| 80 | DM-312 | Come up with a standard to handle C++ Exceptions in Qserv (and the rest of DM?) | Currently CSS is using one class and relies on different error codes to differentiate between different type of exceptions, while other parts of Qserv core define an exception for each different error. It'd be good to standardize and use the same approach. | 4 |
| 81 | DM-313 | cleanup includes in Qserv core modules | Includes need cleanup: group into standard lib, boots and local, sort as appropriate etc. Also, unify forward declarations. | 2 |
| 82 | DM-318 | CSS - surviving mysql and zookeeper glitches | CSS should gracefully recover from failures such as lost connection to mysqld or zookeeper. It is not clear if it would survive such glitches right now -- this needs to be tested and the code improved as necessary. | 4 |
| 83 | DM-319 | Create a board (virtual or otherwise) with pictures and names of everyone in DM | I tried compiling a list of everyone who works or has worked on the DM code -- it was nearly impossible. We should have a (public) list both of current staff and our alumni (and where they are now). | 1 |
| 84 | DM-321 | Re-think thread.cc and dispatcher.cc python interface | The mess of thread.cc and dispatcher.cc need to be re-thought and re-designed so that the interface is smaller and more obvious. | 2 |
| 85 | DM-322 | Trim python importing by czar in app.py | Clean up the way modules are imported in qserv master, use relative import when appropriate instead of lsst.qserv.master.<package> (migrated from Trac #2369) | 2 |
| 86 | DM-326 | Libraries being built in lib64 on OpenSUSE, when EUPS tables assume lib | A report from Darko Jevremovic <darko@aob.rs>: {quote} Hi Mario, I managed to build stack v8 on OpenSuse13.1 There were standard problems with lib/lib64 - namely system builds libraries in $PREFIX/lib64 and some programs are hard wired for $PREFIX/lib if you could change the last line of mysqlclient-5.1.65+3/ups/eupspkg.cfg.sh from (cd $PREFIX/lib && ln -s mysql/* . ) to ( cd $PREFIX && if [ ! -f "lib" ] ; then ln -sf lib64 lib; fi &&cd $PREFIX/lib && ln -s mysql/* . ) or something along that line (am not sure whether the syntax would work). Also if you could add in the same manner to ups/eupspkg.cfg.sh ( cd $PREFIX && if [ ! -f "lib" ] ; then ln -sf lib64 lib; fi) for the following packages: minuit2 gsl cfitsio wcslib {quote} | 1 |
| 87 | DM-327 | Take RAM into account when computing NCORES to use in installs | Darko Jevremovic reported he's had to switch off hyperthreading and manually override NCORES, MAKEFLAGS and SCONFLAGS because his 8-core machine had too little RAM to build afw with -j 8. To fix this, eupspkg default build routines should take RAM into account when computing the level of build parallelism. In the meantime, we should document the workaround (contact darko@aob.rs). | 1 |
| 88 | DM-330 | Local lsst-build invocations should use a different build number prefix | Buildbot-invoked lsst-build installs packages in the stack with "b#" tags. These are propagated to the distribution server by {{eups distrib create}}. Local stacks maintained with lsst-build should use different tags so that they don't conflict with these distributed tags. | 1 |
| 89 | DM-331 | S14 Enhancements in Qserv installation procedure | NULL | 15 |
| 90 | DM-334 | Cut Qserv release | It'd be very useful to have fully functioning Qserv release with the latest set of changes (build, packaging, CSS, Daniel's fixes etc) during the Hackathon week. | 2 |
| 91 | DM-335 | Migrate std::lists to std::vectors | Suggested by Andy when reviewing DM-296, discussed at Qserv mtg 3/27. std::list --> std::vector * why? Default now is vector, iterating over vector is much more efficient than over list * revisit on case by case bases, do not blindly replace * preferred solution: typedef, and name it in a way that conveys the intent (e.g., might call it a "container"), underneath use vector | 8 |
| 92 | DM-336 | improve code that initializes shared_ptrs | Reported by Andy when reviewing DM-296. Discussed at Qserv mtg 3/27. boost::shared_ptr =(new T())" --> boost::make_shared() | 4 |
| 93 | DM-337 | removed dead code in stringUtil.h | Remove obsolete strToDoubleFunc (and more) in util/stringUtil.h. | 1 |
| 94 | DM-339 | Make it easy to build and release point releases | NULL | 4 |
| 95 | DM-342 | Establish github mirror of LSST repositories | NULL | 4 |
| 96 | DM-343 | Transition to Stash | NULL | 10 |
| 97 | DM-345 | Enable gravatars | Could you enable gravatars for all our atlassian products (at least Jira + Agile; Confluence) https://confluence.atlassian.com/display/AOD/Configuring+Gravatar+support | 1 |
| 98 | DM-354 | Add cameraGeom overview to Doxygen documentation | The CameraGeom package needs an overview page (part of afw's main.dox) as part of the Doxygen documentation. I think it's up to Simon or me to add this. | 2 |
| 99 | DM-355 | Install and tag multiple Qserv versions on the same distserver | Done in DM-366 | 1 |
| 100 | DM-359 | Simplify Co-add example in Software User Guide | The current example in the LSST Software User Guide for co-addition reflects the processes necessary to perform a DR production. While thorough, it only really works on the lsst cluster. The example should be simplified to work on a smaller subset of data, and on single-user machines. Definition of Done: * Following the documentation, it will be possible for users to identify the SDSS data they need (the subset of files) * There will be instructions on how to download the necessary files to their local machine * There will be instructions on how to build the necessary repositories * There will be instructions on how to run the Co-Add+forced photometry tasks. * Any issues requiring access rights to LSST machines or databases will be identified and issues created for later. | 8 |
| 101 | DM-365 | Integration tests dataset should be packaged in eupspkg | A qserv-integration-tests package should be created : - it would allow to manage easily, in ups/qserv.table, tests version for a given Qserv version. - it would allow to install Qserv dependencies related to testing, like partition (and other data ingest code which may arrive. | 3 |
| 102 | DM-366 | Refactor install/distribution procedures using lsst-sw | Here's Andy Salnikov remark, during #3100 review : https://dev.lsstcorp.org/trac/ticket/3100#comment:18 | 5 |
| 103 | DM-367 | CSS performance optimizations (avoiding redundant checks) | Facade.cc: it seems worthwhile to think about how to tweak the implementation to reduce redundant calls. e.g., getChunkedTables() calls _cssI->exists() for the database for each contained table. It also calls exists() for each table, which it just retrieved. In reality, it only needs to call exists() once, for the db. So we are doing 1+3t reads rather than 1+t. (This came up in the review of DM-56, the review comments are captured in DM-225) | 8 |
| 104 | DM-368 | integrate qserv_admin backend into czar or separate admind | client/qserv_admin_impl.py: I know the czar's Python code is ugly. But the concept of having more than one entry point for users is also ugly. I feel like the client should just wrap up stuff into JSON and make a REST call into the czar. Ideally the proxy would use REST, perhaps via: https://github.com/fperrad/lua-Spore/, which at least seems more thought-out than lua-xmlrpc. (This came up in the review of DM-56, the review comments are captured in DM-225) | 15 |
| 105 | DM-369 | Improve how CSS exceptions are handled | CSS has one class for all exceptions. The model adopted by Daniel (each type as a separate exception) is better, as we can catch individual exceptions instead of (a) having to catch all css exceptions, (b) checking type and (c) rethrowing if it is not the type we want. To be able to catch all CSS exceptions, we can just introduce a new base class (that is new comparing to Daniel's version). | 6 |
| 106 | DM-370 | improved how default values for CSS are handled | Need to improve how defaults are handled in qserv_admin. There seems to be some desire to warn when values are not set--how about setting defaults and just printing what configuration is being used? If this is something human-created, we should have reasonable defaults and not bother the user, unless no default is viable. I think we should only be strict on machine-generated input, where we would like to catch bugs as soon as possible. (This came up in the review of DM-56, the review comments are captured in DM-225) | 5 |
| 107 | DM-372 | fix testQueryAnalysis | 5 tests fail in the testCppParser. | 2 |
| 108 | DM-373 | Catch AttributeError problems in czar | I wonder if we could catch more exceptions in czar to simplify debugging. For example, if I change: {code} --- a/core/modules/czar/lsst/qserv/master/app.py +++ b/core/modules/czar/lsst/qserv/master/app.py @@ -418,7 +418,7 @@ class InbandQueryAction: self.constraints.size()) dominantDb = getDominantDb(self.sessionId) dbStriping = getDbStriping(self.sessionId) - if (dbStriping.stripes < 1) or (dbStriping.subStripes < 1): + if (dbStriping.x < 1) or (dbStriping.y < 1): {code} It will return to client a very cryptic error: {code} Qserv error: Unexpected error: (<type 'exceptions.AttributeError'>, AttributeError('x',), <traceback object at 0x969a02c>) {code} with no other clues, traceback or information in the log. | 2 |
| 109 | DM-375 | S14 Improve error handling in all parts of Qserv, report sensible errors to users | Qserv needs to handle error gracefully. No matter what error occurs, it should try to automatically recover, and if it can't it should report a reasonable error to user. We should try to poke around and trigger various errors in random places in Qserv and watch what happens, how system fails and/or how it reports the error to user (then we should fix it...). I expect we will need to design a better error handling framework to achieve graceful error handling. | 60 |
| 110 | DM-380 | loadLSST bug(s) for csh, ksh | A flaw in the v8.0 loadLSST scripts (and/or in eups/bin/setups) causes the following errors: 1) When using ksh: {code} $INSTALL_DIR/loadLSST.ksh ksh: /path/to/INSTALL_DIR/eups/bin/setups.ksh: cannot open [No such file or directory] {code} And indeed, there is no eups/bin/setups.ksh file. 2) When attempting to run the installation demo (v7.2.0.0): {code} $> printenv SHELL /bin/tcsh {code} [The same issue appears with csh, unsurprisingly.] {code} $> source /path/to/install_dir/loadLSST.csh $> cd /path/to/demo $> setup obs_sdss $> ./bin/demo.sh ./bin/demo.sh: line 7: /volumes/d0/lsst/stack80/eups/*default*/bin/setups.sh: No such file or directory ./bin/demo.sh: line 12: setup: command not found {code} After hand-editing the demo.sh script to omit the "/default" string from the offending line, the demo runs normally to completion. Note that everything works fine for bash with v8.0, which is what I tested awhile back. | 1 |
| 111 | DM-384 | Add Versioning to SourceTable in lsst::afw::table | Add version to afw::table::SourceTable. Persist that version number to fits file when the table is saved, and restore when the table is restored. Tables created and saved to disk prior to this modification will have the version number 0, by default. Tables created with the S14 version will have the version number 1. This change is to enable a new version of slots and field naming conventions as needed by the Measurement Framework overhaul, at the same time allowing current clients of SourceTable to continue to function. The work to define and persist the slots depending on the version will be on a separate issue. Should not appear as an alterable member of the metadata, but should be saved with the metadata and reloaded when the file is reloaded. getVersion and setVersion methods will be used to allow clients to alter this number. | 1 |
| 112 | DM-386 | Create Command/Event Sender | Simulate the OCS and CCS (via the OCS) sending message to the Base DMCS. Write a library to send commands via method calls, which will be used by commandable entities and by the Base DMCS. The method calls in this library will be used to simulate sending commands from the OCS. This will initially be developed using DM messages, and later switched to use DDS. Write a command line tool to send these messages. Commands with no arguments: init, enable, disable, release, stop, abort, reset. Commands with arguments: configure - arguments are: Set of computers, software and versions to be executed, parameters used to control that software. Events with arguments: startIntegration, nextVisit. Definition of done: * A library that has method calls to each of these commands/events. Each method call sends one message to the given Topic. * Command line tool that can send any of these commands/events to a commandable entity subscribed to a Topic. * Unit tests for each of the commands/events | 6 |
| 113 | DM-387 | Build Base DMCS communications library | Write a library to be used by each commandable entity and the Base DMCS. Methods in this library receiving commands from the simulated OCS. Specific command actions will be handled by each entity, and events are handled by the Base DMCS. This will initially be developed using DM messages, and later switched to use the DDS. The library will include: * An object with methods for blocking receives, blocking receives with timeouts, and non-blocking receives. Any commands received by these methods are given to another class to call appropriate action methods. * An abstract class implementing each of the following methods for commands: init, configure, enable, disable, release, stop, abort, reset. * An abstract class with methods for implementing the following methods for events: startIntegration and nextVisit | 6 |
| 114 | DM-388 | Build replicator | Replicator On boot: * Establishes connection with single, pre-configured distributor. * Checks connection with the network outage buffer, the Base raw image cache, and the tape archive. On success, the replicator registers itself with the Orchestration Manager in the fully-operational pool. From K-T: Note that what to check should eventually be dependent on what was specified in the configuration. That way, if the tape library is down, we could still run using the network outage buffer or raw image cache as the redundancy source. In operation: * Receives a job with visit id, exposure sequence number within the visit and raft id. * Queries the Base EFD replica for information needed to process the image. * Subscribes to the Camera Data System (CDS) “startReadout” event and to the OCS telemetry topics. * Waits for a startReadout event * Request the cross-talk corrected exposure for the raft using the CDS client interface and block until it is available. On receipt of crosstalk-corrected image: * Verify integrity of image (hash or checksum) * Send image with telemetry to the distributor, compressing it if the configuration says to do so. At the same time, the image is written to the network outage buffer and the raw image cache using the Data Access Client Framework, retrying until successful for a configured number of tries. All images that are written are tagged with the Archiver mode (by the replicator job). * Request the raw image using the CDS client interface and block until it is available. On receipt of the raw image: * Verify integrity of image (hash or checksum) * Send image to the distributor and simultaneously write to the network outage buffer and the tape archive. ALL image transmission statuses (successful and unsuccessful) are recorded to the Alert Production Database. NOTE: Image some calibration or engineering modes, there may only be raw-image data, not crosstalk-corrected image data. The replicator job configuration will provide for this). NOTE: Replicator jobs will need to detect they are not completing in the expected amount of time. NOTE: Something (the Base DMCS? The Event/Message Monitor?) needs to pay attention to the number of replicator re-registering themselves with the local-only pool and notify NetOps to investigate and resolve the issue. On errors sending or keeping connected to the distributor, the replicator: * unregisters from the “fully-operational” pool * registers with the “local-only” pool The replicator will continue to try and connect to the distributor, and if it is able to reestablish the connection: * unregisters from the “local-only” pool * registers with the “fully-operational” pool Heartbeat messages between replicators and distributors will indicate when the replicators will be able to re-register with the fully-operational pool. | 10 |
| 115 | DM-389 | Build distributor | Distributor On boot: * Checks the network, archive raw image cache and the tape archive. After all have tested successfully, the distributor waits for a connection from its associated replicator. In operation: * On receipt of a visit id, exposure sequence number and raft id from its associated replicator, the distributor publishes them along with its network address to the Archive DMCS. On receipt of crosstalk-corrected image and associated telemetry from its associated replicator: * Verifies integrity using a hash or checksum * Writes to the raw image cache using the Data Access Client Framework * Decompresses (if necessary) * Separates into individual CCD-sized portions * Sends portions to the appropriate connected Workers Note: Workers can connect to request a CCD-sized crosstalk-corrected image. On receipt of raw image: * Writes it to the tape system. * All images are tagged with the Archive mode. Should include heartbeat monitoring of the replicator/distributor connection. | 10 |
| 116 | DM-405 | Write Linux Standard Base - compliant init.d scripts | Qserv services init.d scripts have to rely on LSB, in order to work on multiple systems. Remark : xrootd has to be launched as a background process (i.e. with a & at the end). But this always send of return code equal to 0, even if xrootd fails to start, a shell function wait_for_pid will be implemented in xrootd init.d scritps to solve that (inspired from mysqld init.d script). | 5 |
| 117 | DM-413 | Simulate computation and production of VOEvents for worker batch job | Simulate jobs which do alert processing. Produce VOEvents based on those results. | 6 |
| 118 | DM-417 | finish adding aliases to afw::table::Schema | This issue picks up the partially-completed Trac Ticket #2351, which adds a string-substitution-based alias mechanism to afw::table::Schema. There are still some issues to sort out w.r.t. constness - in other respects, a Table or Catalog's schema cannot be changed once the Table has been constructed, but we do need to be able to change aliases after table construction. | 6 |
| 119 | DM-419 | Use aliases in slots | NULL | 2 |
| 120 | DM-420 | Remove measurement code from meas_algorithms | NULL | 4 |
| 121 | DM-421 | add basic FunctorKeys | Add the basic FunctorKeys mechanism, and enough implementations to support Slots. | 4 |
| 122 | DM-422 | Add FunctorKeys to replace compound field functionality | Add more FunctorKeys to replace the functionality in all current compound field types and meas_base result objects. May involve moving some meas_base definitions to afw. | 4 |
| 123 | DM-423 | Add FunctorKeys for common analysis tasks | Add FunctorKeys for simple, common, calculated fields, including: - Magnitudes from fluxes - Coords from Points, Points from Coords - Ellipse conversions and radius/ellipticity extraction | 4 |
| 124 | DM-424 | Integrate FunctorKeys with SchemaMapper | NULL | 10 |
| 125 | DM-425 | Remove support for compound field types | NULL | 2 |
| 126 | DM-428 | add live DS9-based debugging to measurement framework | The old measurement framework had a lot of live DS9-based display options. We should ensure the new one has at least as many, and that it still works. At some point, we should consider a different mechanism for enabling those displays and possibly other tools for displaying them, but that's out of scope for this issue. | 6 |
| 127 | DM-429 | Make NoiseReplacer outputs reproduceable | We need a way to get back the noise-replaced Exposure as it was when a particular source was measurement, after the measurement has been run, without having to run noise-replacement on all the previous objects again. There is already code in afw::math::Random to output its state as a string; I think we should probably just save this string in the output catalog. This will require some API changes to allow the NoiseReplacer to modify the schema and set a field in the output records. | 3 |
| 128 | DM-430 | Control log levels on a per-plugin basis | We should be able to control log levels so that certain plugins are run at one level while the rest are run at another (to allow a particular plugin being debugged to be more verbose). | 4 |
| 129 | DM-433 | Add slot support for meas_base-style outputs | The slot mechanism in afwTable currently uses compound fields to save the 3 slot types: flux, centroid, and shape. Since the new measurement framework uses a flattened representation in the SourceTable where these types are saved as multiple scalar fields, the slot mechanism need to be altered to handle this new table type. 1. An alternative to KeyTuple for storing the keys required by the slot 2. Fixup get(Centroid, Flux, Shape) in SourceRecord to use correct keys. 3. Fixup the single value getters (getX, getY, etc) to use the correct keys. 4. Persist slot info to fits correctly, based on table version. | 4 |
| 130 | DM-435 | add aperture-correction measurement code to the end of calibrate | At the end of CalibrateTask, we'll want to compute the PSF and aperture fluxes of the PSF stars, and send those to the PSF model to be stored and interpolated (using the featured added via DM-434). We'll also need to run any other flux measurement algorithms that need to be tied to the PSF fluxes on these same stars; because these can be somewhat slow, we probably want to limit these measurements to only the PSF stars, rather than requiring all these algorithms to be run as part of calibrate.measurement. The relationships between these fluxes and the PSF fluxes will be additional fields to be added to and interpolated by the PSF. The HSC implementation of this work (as well as that of DM-436) was done on issue HSC-191: https://hsc-jira.astro.princeton.edu/jira/browse/HSC-191 There were changes to many packages, but the relevant ones for LSST are: https://github.com/HyperSuprime-Cam/afw/commit/057fb3c0581c512d5664f1883a72da950c9eae9d https://github.com/HyperSuprime-Cam/meas_algorithms/compare/HSC-3.0.0...u/jbosch/DM-191 https://github.com/HyperSuprime-Cam/pipe_tasks/compare/4c3a53e7238cbe9...u/jbosch/DM-191 | 8 |
| 131 | DM-436 | apply aperture corrections in measurement tasks | We need to interpolate the aperture correction to the position of every source, and apply this correction to all appropriate fluxes. The HSC implementation of this work (as well as that of DM-435) was done on issue HSC-191: https://hsc-jira.astro.princeton.edu/jira/browse/HSC-191 There were changes to many packages, but the relevant ones for LSST are: https://github.com/HyperSuprime-Cam/afw/commit/057fb3c0581c512d5664f1883a72da950c9eae9d https://github.com/HyperSuprime-Cam/meas_algorithms/compare/HSC-3.0.0...u/jbosch/DM-191 https://github.com/HyperSuprime-Cam/pipe_tasks/compare/4c3a53e7238cbe9...u/jbosch/DM-191 Note that on the LSST side, we'll want to apply the aperture corrections either within a new plugin in meas_base or as a new part of BaseMeasurementTask, not as a change to the CorrectFluxes algorithm (which will be removed in the future along with the rest of the old measurement framework in meas_algorithms). | 7 |
| 132 | DM-441 | Setup of four new measurement algorithms for processCcd testing | The goal of this story is to take the following algorithms and make them fully operational with processCcd. PsfFlux,SdssShape,SdssCentroid,SincFlux. This code was moved to meas_base in w14, but has yet to be used in full operation. This is the first step in that process. Each algorithm will at least have one test to confirm that it works, but the unit tests will be very simple. The actual confirmation of full operability will the to run tests against real data, and compare against the existing meas_algorithms. The algorithm code will still require cleanup even after this story is completed. That is because it has intentionally been left the same as what existed in meas_algorithms. The goal of this ticket is to confirm that the each algorithm can (1) complete its measurement with some reasonable result, (2) responds to its major configuration options, and (3) handles at least some of its defined exceptions (that is, flags) correctly. Confirmation that the results are identical with meas_algorithms is a subsequent ticket Cleanup and documentation of this code will be done in a subsequent ticket. | 4 |
| 133 | DM-443 | Approve/Acquire HipChat licenses | -------- Original Message -------- Subject: Getting HipChat licenses Date: Wed, 09 Apr 2014 08:51:46 -0700 From: Mario Juric <mjuric@lsst.org> Organization: Large Synoptic Survey Telescope Inc. To: Jeffrey Kantor <JKantor@lsst.org>, Iain Goodenow <IGoodenow@lsst.org>, Stefan Dimmick <SDimmick@lsst.org> Jeff et al., I'd like to start the process to acquire HipChat licenses (we have 10 days left on our evaluation one). Right now we have 26 users, so our first total would be $52/month. I'm expecting it will eventually grow to ~50 users or so. In terms of an account to charge, this should really be considered a project-wide tool, but if that's going to cause unnecessary delays I'd propose we charge it to DM as a sign of our infinite kindness and good will :). HipChat wants to simply bill a credit card -- once we get the necessary approvals, Iain, I can add you as an admin, and you can use our corporate one if that's OK. | 1 |
| 134 | DM-444 | Write job ads for Tucson DM positions | NULL | 4 |
| 135 | DM-445 | Test four algorithms for compatibility with original meas_algorithms | Do a trial run of a small area of sky (using a single exposure from measMosaicData). First create a source catalog using the old measurement.py, then run the same test with the measurement task in sfm.py. Compare the results. If the code has been ported correctly, they should match. | 4 |
| 136 | DM-446 | Setup PeakLikelihoodFlux with new Algorithm Framework | Move PeakLikehoodFlux to meas_base framework | 1 |
| 137 | DM-447 | Setup Flux algorithms for testing with processCcd | Similar to DM-441 for flux algorithms GaussianFlux and NaiveFlux Unit tests for major config options, just to be sure that they do something reasonable. Test of at least one exception (flag) | 2 |
| 138 | DM-448 | Test Centroid algorithms against meas_algorithms | NULL | 2 |
| 139 | DM-449 | Test Flux algorithms agains meas_algorithms | Test for compatibility of NaiveFlux, GaussianFlux, and PsfFlux against meas_algorithms | 2 |
| 140 | DM-454 | reimplement shapelet PSF approximations | The CModel code we want to transfer from HSC in DM-240 currently relies on the old "multishapelet.psf" algorithm in meas_extensions_multiShapelet. That means we either need to convert that PSF-to-shapelets code in meas_extensions_multiShapelet to use the meas_base interface, or we need add new PSF-to-shapelets code in meas_multifit. I think the latter is the better choice, even if we delay DM-240 as a result; the heavy algorithmic code is already available as primitives in meas_multifit, so it should just be a matter of packing those into a simple driver, creating a config class for it, and testing it on a few real and simulated PSFs to learn reasonable defaults for the configs. | 6 |
| 141 | DM-460 | Implement backup/restore for CSS | It'd be very useful to have a way to "dump" and "restore" entire contents of the key/value store in zookeeper. The dump part is pretty much there (can dump to an ascii file) | 4 |
| 142 | DM-461 | Add Classes of MeasurementError | Some measurement failures are global for a whole exposure, such as a missing Psf or Wcs. The framework currently does not distinguish this from a failure in a single measurement. Should add a new subclass of MeasurementError which can be thrown in these cases. Should also add configuration option to the measurement framework to determine what should be done with this type of error. We should also add an exception class and associated how-to-handle config for problems that indicate that something has gone wrong in pre-measurement processing, such as NaNs in the image. | 4 |
| 143 | DM-463 | PixelsFlags, SkyCoordAlgorithm, and Classification | SkyCoord was moved to DM-441 - done in Python Classification is also simple if done in Python Pixel Keys will be done in C++ Some work left from SdssShape will be done with this ticket Also, since these are out only two Python algorithms in the base set, I will add the exception handling and base fail() methods at this time. | 2 |
| 144 | DM-464 | add and use "suspect" flag in slots and slot-like measurements | We need to have both "fatal" and "suspect" flags to handle different levels of warnings in measurement. (other tasks previously included on this ticket have been split into other tickets, including DM-461 and DM-984) | 4 |
| 145 | DM-466 | lsst-build updates based on feedback from 1st month of use | change lsst-build interface to: * rename 'prepare' to 'clone' * use current directory to clone to * remove the default REPOSITORY_PATTERN, as it isn't complete * use the current username as default build tag prefix * don't write the manifest by default; use --manifest=<filename> option instead * refuse to change/overwrite repositories if they're dirty, use --force to override * create a separate 'version' verb * ProductFetcher.fetch doesn't need to return ref, sha1 * read config file within build directory, .btconfig * rename the binary to bt * have the versioner use git db by default, but fall back to generating versions from hash by default See the linked web page for a mock of the command line interface. | 6 |
| 146 | DM-467 | Understand galfast bugs | NULL | 2 |
| 147 | DM-468 | Alias measurement.plugins to measurement.algorithms | The config item in the old measurement task, measurement.algorithms was changed to measurement.plugins in meas_base. The creates a backward compatibility issue for code which refers to this class member. Jim's suggested fix is to alias plugins with algorithms in the new measurement task. | 1 |
| 148 | DM-470 | Rework exceptions in css (python side) | Rework exceptions in css/KwInterface.py: split into key-value related exceptions, possibly moving the rest that deals with db/tables into client. This came up in the CSS review, see DM-225: "CssException feels a bit out of place...." | 1 |
| 149 | DM-471 | Create LSSTsim processing example for SW User Guide | Create a worked example in the Software User Guide of processing raw data with obs_lsstSim. The example should include: - Pointers to documents for creating multi-band, raw datasets with PhoSim - Creating a data repository for the data - Creating an astrometry_net_data repository - Processing raw files through processCdc (including ISR, measurement, etc.) - Creating a catalog of measurements - Visualizing the output using existing tools It is possible to work initially with a limited set of data (a raft, or CCD), and to start with eImages rather than raw; the example can be elaborated in due course. | 10 |
| 150 | DM-472 | Create an iPython Notebook visualization of the LSST Demo Data | Develop an iPython Notebook to illustrate the processing and results of the LSST Demo. This can be used as both a basic tutorial for the Stack, and a how-to for creating additional visualizations for DM. The visualization should include the source catalog, and possibly also one or more of the processed images. | 4 |
| 151 | DM-473 | Prioritize and define the backlog for Summer 2014 | At the DMLT meeting in Seattle, finish the backlog prioritization for Summer 2014 cycle. Definition of Done: * All epics and major stories defined for Summer 2014. * Rank-ordered in terms of priority. * Leadership teem agreement on their prioritization and backlog. * Teams identified to execute the stories, and stories labeled accordingly. | 2 |
| 152 | DM-474 | Document how to create an astrometry_net_data repository | Document how to create astrometry_net_data index files for an arbitrary astronomical dataset, as well as how to create a repository of such data to support processing with the LSST Stack. It must be possible to limit the area of sky coverage in the index files to that appropriate for the images that the user wishes to process. | 6 |
| 153 | DM-488 | Make JIRA notification e-mail more useful | From HipChat/Data Management: [12:09] Mario Juric: @jbosch @KTL @KSK Could you double-check if any of you got an e-mail from Jira on Saturday (Apr 12th) re issue DM-78 (I made you reviewers, but it looks like you weren't notified)? [12:10] K-T Lim: I don't recall and can't determine now; it would have been deleted (irrevocably). [12:10] Simon Krughoff: I did get an email. [12:10] Jim Bosch: @mjuric, ah, it appears that I actually did. The fact that I was a reviewer was just buried, and I didn't notice it. [12:10] Simon Krughoff: I must have missed that I was a reviewer. [12:10] Mario Juric: OK, thanks! That gives me not one, but two useful data points (#1 -- emails work, #2 -- they're useless :) ). [12:12] Simon Krughoff: I'm not sure why they are useless. The emails from trac were a very important part of my workflow as far as being notified of review responsibility goes. Maybe it's just the volume from Jira. [12:14] Jim Bosch: Yeah, same here. Though the volume from JIRA hasn't been so bad, so I don't think that's it. Maybe my brain just has to get used to the new email format. [12:14] K-T Lim: (In my case, I'm mostly paying attention to the RSS feed although the mailbox serves as a backup.) [12:22] Robert Lupton: One of the things that made gnats a good bug tracker was that the emails contained the right amount of information (I did have source code...), and trac was pretty good too when we tuned it; bugzilla always used to be awful. I bet we can fiddle with Jira to make its mail more useful; I don't just mean filtering what it sends, but making sure that each email is self contained, but not too long | 1 |
| 154 | DM-495 | Build Base DMCS Archiver Command Receiver | Base DMCS Archiver The Archiver receives commands from the OCS, and performs the following actions: * init - Move from OFFLINE mode to IDLE state. * configure - (arguments: set of computers to use, the software to be executed, and parameters to control that software). 1) Verify format and accept command; 2) change to disable mode as if a disable command had been received; 3) install configuration; 4) return success to the OCS. On failure (illegal configuration, cannot be installed properly), a command error with the failure reason is sent to the OCS. prerequisite for configure command - sufficient replicator/distributor pairs are available. Note that after configure, the Archiver remains disabled. * enable - Subscribe to the startIntegration event. * disable - Unsubscribe to the startIntegration event. Note that this does NOT terminate any replicator jobs which are already executing. * release - Unsubscribe to the startIntegration event, and go into OFFLINE state. * abort - if received during a configure command, causes this component to go into the ERROR state with no configuration. If received at any other time, the system transitions into an ERROR state, but nothing stops that was already in progress. * reset - Unsubscribe to the startIntegration event, go into IDLE state, with no configuration | 6 |
| 155 | DM-502 | Build Base DMCS Catch-Up Archiver Command Receiver | Base DMCS Catch-Up Archiver The Catch-Up Archiver receives commands from the OCS, and performs the following actions: * init - Move from OFFLINE mode to IDLE state. * configure - (arguments: set of computers to use, the software to be executed, and parameters to control that software). 1) Verify format and accept command; 2) change to disable mode as if a disable command had been received; 3) install configuration; 4) return success to the OCS. On failure (illegal configuration, cannot be installed properly), a command error with the failure reason is sent to the OCS. Prerequisite for configure command - sufficient catch-up-dedicated replicator/distributor pairs are available. Note that after configure, the Archiver remains disabled. * enable - 1) Allows Catch-Up Archiver to scan for unarchived images to be handled; 2) Enables the Orchestration Manager to schedule image archive jobs. * disable - 1) Stop scanning for unarchived images; 2) Tell the Orchestration Manager to stop scheduling any new image archive jobs. * release - Unsubscribe to the startIntegration event, and go into OFFLINE state. * abort - if received during a configure command, causes this component to go into the ERROR state with no configuration. If received at any other time, the system transitions into an ERROR state, but nothing stops that was already in progress. * reset - Unsubscribe to the startIntegration event, go into IDLE state, with no configuration | 6 |
| 156 | DM-503 | Build Base DMCS EFD Replicator Command Receiver | Base DMCS EFD Replicator The EFD Replicator receives commands from the OCS, and performs the following actions: * init - Move from OFFLINE mode to IDLE state. * configure - (arguments: set of computers to use, the software to be executed, and parameters to control that software). 1) Verify format and accept command; 2) change to disable mode as if a disable command had been received; 3) install configuration; 4) return success to the OCS. On failure (illegal configuration, cannot be installed properly), a command error with the failure reason is sent to the OCS. Prerequisite for configure command - communication with the US DAC EFD replica is possible. Note that after configure, the EFD Replicator remains disabled. * enable - Causes the Base DMCS to enable the US DAC EFD replica to be a slave to the Chilean DAC EFD replica. * disable - Causes the Base DMCS to disable the slave operation of the US DAC EFD replica. * release - Unsubscribe to the startIntegration event, and go into OFFLINE state. * abort - if received during a configure command, causes this component to go into the ERROR state with no configuration. If received at any other time, the system transitions into an ERROR state, but nothing stops that was already in progress. * reset - Unsubscribe to the startIntegration event, go into IDLE state, with no configuration | 6 |
| 157 | DM-504 | Build Base DMCS Alert Production Cluster Command Receiver | Base DMCS Alert Production Cluster The Alert Production Cluster receives commands from the OCS, and performs the following actions: * init - Move from OFFLINE mode to IDLE state. * configure - (arguments: set of computers to use, the software to be executed, and parameters to control that software). 1) Verify format and accept command; 2) change to disable mode as if a disable command had been received; 3) install configuration; 4) return success to the OCS. On failure (illegal configuration, cannot be installed properly), a command error with the failure reason is sent to the OCS. Prerequisite for configure command - sufficient workers are available. Note that after configure, the Alter Production Cluster remains disabled. * enable - Causes the Base DMCS to subscribe to the “nextVisit” topic in normal science mode; another event may be subscribed to in calibration or engineering mode. * disable - Causes the Base DMCS to unsubscribe from the “nextVisit” topic. It does NOT terminate any worker jobs already executing. In particular, the processing for the current visit (not just exposure) will normally complete. * release - Unsubscribe to the startIntegration event, and go into OFFLINE state. * abort - if received during a configure command, causes this component to go into the ERROR state with no configuration. If received at any other time, the system transitions into an ERROR state, but nothing stops that was already in progress. * reset - Unsubscribe to the startIntegration event, go into IDLE state, with no configuration | 6 |
| 158 | DM-505 | improve initialization of kvMap in testQueryAnalysis | Build the kvMap at build-time and embed it into the executable. (this was brought up in DM-225) | 1 |
| 159 | DM-506 | improve generating kvMap in testFacade.cc | Generating the kvmap file, and pasting it into a string inside the test program. (this was brought up in DM-225) | 1 |
| 160 | DM-508 | shorten internal names in zookeeper | rename DATABASE_PARTITIONING to PARTITIONING rename DATABASES to DBS | 2 |
| 161 | DM-509 | rename "dbGroup" to "storageClass" in CSS metadata | It is meant to be used to indicate L1, L2, L3... At Qserv design week we decided to rename it (original plan was to remove it all together) | 1 |
| 162 | DM-510 | Tweak metadata structure for driving table and secondary index | There seem to be confusion about driving table and secondary index. At the moment in zookeeper structure we have {code} /DATABASES/<dbName>/objIdIndex /DATABASES/<dbName>/TABLES/<tableName>/partitioning/secIndexColName /DATABASES/<dbName>/TABLES/<tableName>/partitioning/drivingTable /DATABASES/<dbName>/TABLES/<tableName>/partitioning/latColName /DATABASES/<dbName>/TABLES/<tableName>/partitioning/lonColName /DATABASES/<dbName>/TABLES/<tableName>/partitioning/keyColName {code} Issues to think about: * we can't call it objIdIndex, it is too lsst-specific. * drivingTable and keyColName - perhaps these should be at database level, which means we would only allow one drivingTable and one secondary index per database? * or, maybe instead of database level, it is a partitioning parameter? Note that two databases might use different name for secondary index or driving table, yet they might be joinable. That argues for introducing a new group, something like /DATABASE/partitioning in addition to /DATABASE_PARTITIONING. * consider renaming drivingTable to keyTable * do we really need secIndexColName and keyColName? Can't we get rid of one, and rename to keyColName? | 2 |
| 163 | DM-511 | rework ubuntu.patch | NULL | 1 |
| 164 | DM-512 | Generalizing data chunking (n-level chunking rather than stripes/subStripes) | It'd be cleaner to use numbering (e.g. 0 for no chunking, 1 for chunks, 2 for subchunks etc) instead of "chunks", "subchunks" throughout qserv code. This might also be true in partitioner, where flags like -S and -s etc are not entirely obvious. This came up in the review of CSS, see DM-225. | 3 |
| 165 | DM-513 | fix threading issues in CSS watcher | Fix problems with threads in watcher.py brought up in DM-225 by Serge: * A thread per database doesn't scale * There is a thread leak when a database is deleted * There is another design problem, in that each database thread looks like it is holding on to the same lsst.db.Db instance under the hood. I don't remember any consideration for thread safety from the lsst.db code when I reviewed it. Note for one that it is not safe to use a MySQL connection simultaneously from multiple threads (and I seem to recall that you are caching a connection inside Db instances). In practice, even the Python GIL may not save you, since calls into C code (i.e. the mysql client library) may very well release it. | 8 |
| 166 | DM-514 | Switch to the "czar" name consistently | 1) Change lsst.qserv.master to lsst.qserv.czar in the czar module. 2) Rename masterLib to czarLib 3) Rename startQserv to startCzar | 4 |
| 167 | DM-516 | Fix race condition when creating db (and elsewhere?) in client/qserv_admin_impl.py | This came up in the review of CSS, see DM-225: QservAdminImpl: so, if there's a problem while inserting the various keys for a database, and another exception during cleanup, you can end up with a partially constructed database. How do you plan on handling this? This is perhaps another argument for trying to batch metadata more - if it's all in a single key, this worry goes away (at least in this particular case). The createDb implementations seem racy. Consider what happens if 2 admins try to create the same database D. Suppose admin A does the _dbExists(D) check first; it returns false. Then admin B comes along, and does the same check, which returns false again. Let's further suppose that admin B successfully issues all the key creates for D without being interrupted by A. Back to A: when A tries to create all the keys required for D, an exception will be raised immediately since D exists. At this point A will dutifully try to clean up (to avoid leaving around a partially constructed D), deleting D as created by B. End result: D was successfully created, no explicit drop was issued, but D does not exist. Database drops can also race with table creates: zookeeper doesn't have a native recursive delete. In other words, it is perfectly possible for an admin A to issue a database drop while someone else is issuing a table create. The recursive delete might descend into the table "directory" while the create is populating it with child znodes. The recursive delete will get an incomplete list of children, remove them, and finally fail to remove the table directory because the table create has since added more children, and it is illegal to remove a non-empty directory. At this stage, the drop will report success even though it has failed (that's the recursive delete behavior that kazoo gives you), and the table create will report success. But the database will still exist, and table keys may be partially present in zookeeper. | 10 |
| 168 | DM-517 | qserv_admin needs to deal with uncommon names/characters | qserv/admin/bin/qserv_admin.py needs to deal with strange names (e.g. with embedded spaces or semi-colons), or at minimum, catch and forbid them. | 3 |
| 169 | DM-518 | Rework exceptions in qserv client | There is a bunch of (I think) unnecessary translation from KvException to QservAdmException. Can't you just handle printing KvException in CommandParser.receiveCommands(), and get rid of the CSSERR error code? (This is in /admin/bin/qserv-admin.py) (this came up in DM-225) | 1 |
| 170 | DM-519 | rethink configuration for client | _fetchOptionsFromConfigFile in client/qserv_admin.py:: If you are going to use the ConfigParser library, please use SafeConfigParser (or RawConfigParser if you don't need string interpolation). But anyway, I'm not sure the INI file format is the right one to use here, as it forces the use of sections in parameter files, which doesn't make much sense (I notice your code just throws away the section names when reading parameter files). I found myself wishing that the admin client just had syntax for the various options, rather than relying on more or less undocumented parameter files. (this came up in DM-225) | 3 |
| 171 | DM-520 | Remove old partitioner/ loader and duplicator | Once Fabrice has migrated the integrated tests towards using the new partitioner and duplicator, we should delete the old partitioner/duplicator (in {{client/examples}}). | 1 |
| 172 | DM-521 | Confusing error message (non-existing column referenced) | A query that references non existing column for non-partitioned table results in a confusing message: "read failed for chunk(s): 1234567890". To repeat, run something like {code} SELECT whatever FROM <existingTable>; {code} Similar error occurs when we try to reference non-existing table, try something like: {code} SELECT sce.filterName FROM StrangeTable AS s, Science_Ccd_Exposure AS sce WHERE (s.scienceCcdExposureId = sce.scienceCcdExposureId); {code} | 5 |
| 173 | DM-522 | gracefully handle misconfigured scons | When I try to run scons using the latest master (89aaa6), it fails with {code} scons: Reading SConscript files ... AttributeError: 'NoneType' object has no attribute 'rfind': File "/home/becla/cssProto/qserv_css6/SConstruct", line 17: state.init(src_dir) File "/home/becla/cssProto/qserv_css6/site_scons/state.py", line 161: _initEnvironment(src_dir) File "/home/becla/cssProto/qserv_css6/site_scons/state.py", line 128: _initVariables(src_dir) File "/home/becla/cssProto/qserv_css6/site_scons/state.py", line 89: (PathVariable('XROOTD_DIR', 'xrootd install dir', _findPrefix("XROOTD", "xrootd"), PathVariable.PathIsDir)), File "/home/becla/cssProto/qserv_css6/site_scons/state.py", line 53: (binpath, binname) = os.path.split(binFullPath) File "/usr/lib/python2.7/posixpath.py", line 83: i = p.rfind('/') + 1 {code} The scripts should check that requires variables are not set, and print appropriate error (and ideally, suggest how to fix it) | 1 |
| 174 | DM-527 | make Image construction robust against integer overflow | I just fixed a bug on the HSC side (DM-523) in which integer overflow in the multiplication of width and height in image construction caused problems. We should backport this fix to LSST. | 1 |
| 175 | DM-530 | Table column names in new parser | Running tests (qserv-testdata.sh) on pre-loaded data I have observed that many test fail for the only reason that the column names in the dumped query results are different between mysql and qserv. Here is an example of query reqult returned from mysql: {code} mysql> SELECT sce.filterName, sce.field, sce.camcol, sce.run FROM Science_Ccd_Exposure AS sce WHERE sce.filterName = 'g' AND sce.field = 670 AND sce.camcol = 2 AND sce.run = 7202; +------------+-------+--------+------+ | filterName | field | camcol | run | +------------+-------+--------+------+ | g | 670 | 2 | 7202 | +------------+-------+--------+------+ {code} and this is the same query processed by qserv: {code} mysql> SELECT sce.filterName, sce.field, sce.camcol, sce.run FROM Science_Ccd_Exposure AS sce WHERE sce.filterName = 'g' AND sce.field = 670 AND sce.camcol = 2 AND sce.run = 7202; +----------+----------+----------+----------+ | QS1_PASS | QS2_PASS | QS3_PASS | QS4_PASS | +----------+----------+----------+----------+ | g | 670 | 2 | 7202 | +----------+----------+----------+----------+ {code} We discussed this already with Daniel yesterday and at qserv meeting today, here I just want to collect what we know so far so that we can return to this again later. As Daniel explained to me this is the result of the new parser assigning aliases to the columns which do not define aliases for themselves. This helps with tracking query proceeding through the processing pipeline. Daniel's observation is that different database engines may assign different names to result columns (or some may not even assign any names), there is no standard in that respect so there is no point in trying to follow what one particular implementation does. Additionally there are issues with conflicting column names and names which are complex expressions. Difference in column names breaks our tests which dump complete results including table header. The tests could be fixed easily, we could just ignore table headers when dumping the data. More interesting issue is that there may be use cases for better compatibility between mysql and qserv including result column naming. In particular standard Python mysql interface allows one to use column names to retrieve values from queiry result. If qserv assigns arbitrary aliases to the columns it may confuse this kind of clients. This issue depends very much on what kind of API qserv is going to provide to clients. If mysql (wire-level) protocol is going to be the main API (which would allow all kinds of mysql clients to talk to qserv directly) then we should probably think more about compatibility with mysql. OTOH if we decide to provide our own API then this may not be an issue at all (but we still need to fix current test setup which is based on mysql). We probably should discuss API question at our dev meeting. | 5 |
| 176 | DM-531 | Qserv returns error table instead of error code | Running the tests on pre-loaded data I noticed that for some queries qserv returns result which does not look like it is related in any way to the query - result column names are different from the columns in the query (different in a different way from DM-530). Here is an example of query and result produced: {code} mysql> SELECT sce.filterName, sce.field, sce.camcol, sce.run FROM Science_Ccd_Exposure AS sce WHERE sce.filterName like '%' AND sce.field = 535 AND sce.camcol like '%' AND sce.run = 94; +---------+------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+ | chunkId | code | message | timeStamp | +---------+------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+ | -1 | 0 | NULL | 1.39779e+09 | | -1 | 100 | Dispatch Query. | 1.39779e+09 | | 32767 | 1200 | Query Added: url=xroot://qsmaster@127.0.0.1:1094//q/LSST/1234567890, savePath=/dev/shm/qserv-salnikov-80df10ee4ed55693702f55021486cd45647c3a58ce549e7c826d9626/7_1234567890_0 | 1.39779e+09 | | 32767 | 1300 | Query Written. | 1.39779e+09 | | 32767 | 1400 | Results Read. | 1.39779e+09 | | 32767 | 1500 | Results Merged. | 1.39779e+09 | | 32767 | 1600 | Query Resources Erased. | 1.39779e+09 | | 32767 | 2000 | Query Finalized. | 1.39779e+09 | +---------+------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+ {code} Daniel and Bill explained to me what is happening there - when query results in an error instead of returning mysql error condition (which is an error code plus some text) proxy produces a diagnostic result which is a table (above) containing some info which could be useful to diagnose the issue. This feature looks potentially useful but it may also be confusing for clients like me who are not aware of this feature. For regular mysql client it may be actually harder to intercept errors because one would need to analyze returned table to understand that error condition happened. It may also be ambiguous in a sense that the legitimate query could produce result with the same column names. Like in DM-530 this issue is tied to a question what kind of API we want to provide to qserv clients. If mysql wire-level protocol is going to be our main API then we should probably try to be more mysql-compatible. OTOH if we are going to hide everything behind our own API then we may have more freedom in re-defining what kind of result error condition produces. | 4 |
| 177 | DM-533 | transfer multiband processing changes from HSC | We have a new multi-band processing scheme for coadds on the HSC side, encompassing changes in afw, meas_deblender, and pipe_tasks. These should be transferred to the LSST side, with RFCs for any backwards incompatible changes. Changes to the Footprint classes may only be temporary, as we plan to refactor those classes soon anyway, but they're still worth doing now to support the higher-level changes. | 8 |
| 178 | DM-536 | Move HSC issues to hsc-jira.astro.princeton.edu | NULL | 10 |
| 179 | DM-545 | ensure pipe_tasks, obs*, and other packages are compatible with meas_base | The switch to meas_base will involve changing the names of most measurement algorithms, as we're using a new naming convention that provides more traceability. This will break downstream code that: - uses field names instead of slots to access measurements - reads or modifies the list of configured-to-run algorithms. Whenever possible, we should fix the former by converting them to use slots, as this will automatically provide backwards compatibility. I'm not yet sure how to handle the latter; the easiest solution would be to give up on full backwards-compatibility with meas_algorithms, but we might be able to find some way to make the old names aliases to the new ones during the deprecation period. | 12 |
| 180 | DM-546 | scons rebuilds targets without changes | I'm seeing something strange when I run scons from current master - running 'scons install' after 'scons build' re-compiles several C++ files even though nothing has changed between these two runs: {code:bash} $ scons build scons: Reading SConscript files ... ... scons: Building targets ... scons: `build' is up to date. scons: done building targets. $ scons install scons: Reading SConscript files ... ... scons: Building targets ... swig -o build/czar/masterLib_wrap.cc -Ibuild -I/usr/include/python2.6 -python -c++ -Iinclude build/czar/masterLib.i g++ -o build/czar/masterLib_wrap.os -c -g -fPIC -D_FILE_OFFSET_BITS=64 -fPIC -I/u2/salnikov/STACK/stack/Linux64/protobuf/2.4.1/include -I/u2/salnikov/STACK/stack/Linux64/xrootd/qs5/include/xrootd -I/u2/salnikov/STACK/stack/Linux64/mysql/5.1.65/include -Ibuild -I/usr/include/python2.6 build/czar/masterLib_wrap.cc g++ -o build/control/AsyncQueryManager.os -c -g -fPIC -D_FILE_OFFSET_BITS=64 -fPIC -I/u2/salnikov/STACK/stack/Linux64/protobuf/2.4.1/include -I/u2/salnikov/STACK/stack/Linux64/xrootd/qs5/include/xrootd -I/u2/salnikov/STACK/stack/Linux64/mysql/5.1.65/include -Ibuild -I/usr/include/python2.6 build/control/AsyncQueryManager.cc g++ -o build/control/dispatcher.os -c -g -fPIC -D_FILE_OFFSET_BITS=64 -fPIC -I/u2/salnikov/STACK/stack/Linux64/protobuf/2.4.1/include -I/u2/salnikov/STACK/stack/Linux64/xrootd/qs5/include/xrootd -I/u2/salnikov/STACK/stack/Linux64/mysql/5.1.65/include -Ibuild -I/usr/include/python2.6 build/control/dispatcher.cc g++ -o build/control/thread.os -c -g -fPIC -D_FILE_OFFSET_BITS=64 -fPIC -I/u2/salnikov/STACK/stack/Linux64/protobuf/2.4.1/include -I/u2/salnikov/STACK/stack/Linux64/xrootd/qs5/include/xrootd -I/u2/salnikov/STACK/stack/Linux64/mysql/5.1.65/include -Ibuild -I/usr/include/python2.6 build/control/thread.cc g++ -o build/merger/TableMerger.os -c -g -fPIC -D_FILE_OFFSET_BITS=64 -fPIC -I/u2/salnikov/STACK/stack/Linux64/protobuf/2.4.1/include -I/u2/salnikov/STACK/stack/Linux64/xrootd/qs5/include/xrootd -I/u2/salnikov/STACK/stack/Linux64/mysql/5.1.65/include -Ibuild -I/usr/include/python2.6 build/merger/TableMerger.cc scons: `install' is up to date. scons: done building targets. {code} This is kind of unexpected, or at least I can't understand now why it happens. Trying to run with --debug=explain shows that some dependencies have disappeared and in some dependencies order is different. No clue yet what that means and how it could happen. Need to study our scons scripts to understand what is going on. | 3 |
| 181 | DM-548 | rearchitect Qserv to fix dependencies between modules in qserv/core | I looked at includes in "core/modules", here is a summary. * ":" indicates a dependency * global, log, util, wbase, wlog and xrootd are low level, it is an easy, I didn't bother showing them below * "!!!" indicates circular dependency * lower-level modules first {core} css: <nothing> global: <nothing> mysql: <nothing> obsolete: <nothing> proto: <nothing> sql: mysql wconfig: sql, mysql wpublish: sql, wconfig !!! wsched <--> wcontrol !!! wcontrol <--> wdb wsched: proto, wcontrol wcontrol: mysql, sql, wsched, wdb, wcontrol, obsolete, proto wdb: sql, mysql, proto, wconfig, wcontrol xrdfs: wpublish, wdb, wconfig, sql, obsolete, wcontrol merger: sql, xrdfs xrdoss: obsolete, wpublish, xrdfs !!! control <--> qdisp !!! qana <--> qproc !!! qana <--> query !!! parser --> query -->qana --> parser !!! query --> qana --> qproc --> query control: css, merger, obsolete, qdisp, qproc, query qdisp: control qana: css, parser, qproc, query qproc: css, merger, parser, proto, qana, qdisp, query query: css, qana, qdisp parser: query {core} | 10 |
| 182 | DM-554 | Base DMCS and Replicator Interaction for Simulator - v1 | Create the Base DMCS and Replicator processes that demonstrate Send startIntegration Send startReadout Assume: Archiver and Alert Production Cluster already enabled, replicators already registered in fully-operational pool. Demonstrate: Sending startIntegration event to the Base DMCS. Replicator jobs submission Replicator jobs subscribe to startReadout Sending the startReadout event to the replicator jobs. Replicator jobs receive startReadout Replicator jobs pull data | 10 |
| 183 | DM-559 | clean up include <> --> "" for third party includes | According to our coding standard 4.15: https://dev.lsstcorp.org/trac/wiki/C%2B%2BStandard/Files#a4-15.OnlysystemincludefilepathsSHALLbedelimitedwith we should be using "" for boost, but in quite a few places we do not: {code} grep 'include <boost' */* |wc 146 314 7916 {code} | 1 |
| 184 | DM-560 | cleanup includes - add module name | Change places like {code}#include "cssException.h"{code} to {code}#include "css/cssException.h"{code} | 1 |
| 185 | DM-561 | Add distributor to simulator - v2 | Add distributor to the simulator Demonstrate: Replicator node associated with to distributor node Replicator jobs send job info to distributor node Replicator jobs copy data to distributor node. | 6 |
| 186 | DM-564 | Add Archive DMCS to simulator - v3 | Add Archive DMCS Demonstrate: Receiving information from duplicator (visit id, exposure sequence number, raft id, and network address). | 4 |
| 187 | DM-566 | Fix Doxygen Doc for new meas_base classes | The final stage of moving the old algorithms to meas_base will be to generate the doxygen documentation and be sure that it is useful. We will do this after the cleanup of the old meas_algorithms code. | 3 |
| 188 | DM-583 | Investigate Approaches to Dcr | This story captures the work of investigating the implications of the different options to compensate for Dcr: ignore objects with extreme colors, compensate for Dcr in measurement, and compensate per-image at the per-object/pixel level. We anticipate the latter option will be required going forward, but we need to do the background work to justify this. We need to understand exactly which classes will need to make use of Dcr information, and what the limitations are of operating at the per-pixel level (e.g. blends). | 20 |
| 189 | DM-584 | Validation of Dcr Approach at the Pixel Level | Before we start the process of extending the Dcr functionality into the stack (e.g. Psf, Wcs, etc), we need to validate that the implementation works at the pixel level. In this task we will work with simulated images to debug and optimize the implementation, to the degree possible. | 30 |
| 190 | DM-586 | Cleanup Source.h.m4 | This include file was damaged somewhat by the addition of slot routines to work with the flattened field definitions. It would be nice to put the Measurement abstraction back in place -- or get rid of it. We need to decide whether the old slot and compound key mechanisms from SourceTable version 0 are going to be continued for doing this. | 2 |
| 191 | DM-587 | The way the observation date is translated by obs_cfht breaks the defect registry. | The fields of the observation date and time as stored in the CFHT-LS headers are not padded with zeros. This makes the sqlite DATETIME constructor return a NULL string when the observation time is < 10hrs. The fix is to force the fields to be padded when the metadata from the input files are ingested. | 1 |
| 192 | DM-588 | Get one HTCondor ClassAds scenario running | We have a worker script that takes three arguments: a CCD number, a cache limit, and a length of time. The script starts out looking at a slot-specific filesystem for a calibration file /local/slot#/calib/CCD#. If that file doesn't exist, it checks to see if there are more than the cache limit of files in /local/slot#/calib. If there are, it removes one at random (later we could try other algorithms). It then immediately transfers /gpfs/calib/CCD# (maybe a 1 GB file) to /local/slot#/calib/CCD#. The script then waits for the given length of time and exits. We make sure we can measure how many file copies from /gpfs are performed and when so that we can determine peak demand rates and average rates. Meanwhile, the HTCondor hook checks for the same /local/slot#/calib/CCD# files and sets the ClassAds based on which CCD#s are present. Scenario 1) 4 slots, 4 CCDs, cache limit 1, jobs take 1 min, total of 5 jobs per CCD, 1 job per CCD submitted every 2 min (1234 pause 1234 pause 1234 pause 1234 pause 1234) | 14 |
| 193 | DM-590 | load non-LSST FITS tables as version 1 | In DM-384 and DM-242, we disabled the periods-to-underscores translations for new tables ("version 1") but left it in place for old tables ("version 0"). In addition, when reading a table without a version number, we assumed it was version 0, to maintain backwards compatibility. I think we should modify this slightly: we should assume a table without a version number is version 0 if and only if it also has the AFW_TYPE key. Otherwise we should assume it is version 1. This will allow us to load externally-produced tables without turning any underscores they contain into periods, while still maintaining backwards compatibility with older tables written by afw. | 1 |
| 194 | DM-593 | Update all DM Software Copyright and License Agreement notices to reflect AURA/LSST | The lsstcorp.org/LegalNotices/{LsstLicenseStatement.txt LsstSourceCopyrightNotice.txt} need to be updated to reference AURA/LSST. The referenced list of LSST partner institutions needs to be either resurrected or the reference deleted. The git repository for devenv/templates needs the Copyright templates to be updated. LsstLicenseStatement.txt needs to be updated to include recent additions of 3rd party tools' Licenses (~10 tools) to the DM stack and all the QSERV 3rd party tools' Licenses (~25 tools). The Copyright banner in all software needs to be updated to reflect the new reality of AURA/LSST in place of LSST Corporation. Files with no Copyright banner, need to add it. Update may occur 'the next time' the code file is updated. THis needs to be broadcast to the developers once the Copyright templates and the website versions are updated. | 2 |
| 195 | DM-594 | running multiple Qserv installations on the same machine | It would be very useful to be able to run multiple installations of Qserv on the same machine (say, 2 developers playing with Qserv on the same machine). I guess we are almost there, we just need to know how to configure all ports so that we are not colliding. Can you test it, tweak whatever is necessary and document what is involved in changing defaults to a unique set of port numbers? | 6 |
| 196 | DM-595 | Setup multi-node Qserv | We are currently focusing on single-node Qserv. It'd be nice to try setting up multi-node Qserv (say 4 workers and a czar on lsst-dbdev*), and improve installation scripts to simplify the process. | 6 |
| 197 | DM-596 | Fix automated tests after css migration | After yesterday's merge of DM-58 into master automated tests do not work any more. The part which is broken now is loading of metadata into qserv. We need to replace old script which created metadata with something different that creates metadata using new CSS. The code which loads metadata in tests is in QservDataLoader class, createQmsDatabase() method (in tests/python/lsst/qserv/test/ directory). | 2 |
| 198 | DM-597 | reorganize client module | move everything in the client package (qserv_admin*, associated tests and examples) to admin/ move css/bin/watcher.py to admin/ | 1 |
| 199 | DM-599 | Investigate off-the-shelf data distribution tools | Research available off the shelf tools, try to find one that would best fit our needs. Produce a document (trac page). | 12 |
| 200 | DM-600 | fix 12 issues in testCppParser (related to switching to CSS) | NULL | 3 |
| 201 | DM-602 | Add support for installing qserv on machines without internet | It is common we install qserv on machines that are on internal network without external network connectivity. How do we do that? | 9 |
| 202 | DM-603 | Look into git-fat | Look into the use of git-fat with the LSST DM workflow. Specifically, how does this work with anonymous access. | 4 |
| 203 | DM-604 | Update parse/analysis tests to detect missing css-kvmap early | Due to the CSS code merge, the testCppParser test depends on an external kvmap file. If this file doesn't exist, nearly every test will fail. There isn't a way to check whether the cssFacade (constructed from kvmap file) is valid. This ticket includes, at minimum: * changes to css/ and qproc/ to make the unit tests fail early if the facade could not be constructed. Optionally, this ticket could include: * renaming testCppParser to testQueryAnalyzer * compile-time linking the kvmap into testCppParser to eliminate the need for an external kvmap file. | 4 |
| 204 | DM-607 | CSS throws exception if tableIsSubChunked is called for non partitioned table | NULL | 3 |
| 205 | DM-608 | referring to table without database context crashes czar | Running a query like "select * from Object" if we do not have database context results in {code} terminate called after throwing an instance of 'lsst::qserv::css::CssException_InternalRunTimeError' what(): Internal run-time error. (*** css::KvInterfaceImplZoo::exists(). Zookeeper error #-8. (/DATABASES/)) {code} Need to gracefully catch the zookeeper -8. Need to detect that empty database name is passed in Facade. Need to avoid it higher up. | 3 |
| 206 | DM-609 | afw unit tests not built unless afwdata available | If afw_data is not available then the afw unit tests are not built. I think it would make more sense to build all of them and run those we can (not all depend on afw_data). One advantage is that a version of afw installed using "eups distrib install" would include built tests, so the tests could be run. This is presently not practical because the SConstruct file is not installed (I intend to open a ticket about that, as well). | 1 |
| 207 | DM-611 | Switch kazoo version to 2.0b1 or later | While we aren't using any of the new features of Kazoo's 2.x series, the removal of zope.interface as a dependency is a worthwhile feature. The 2.0b1 release seems at least as stable as our own code, so I don't think we'll see any negative effects. This ticket covers: - Upgrade of the packaged kazoo from 1.3.1 to 2.0b1 (or later). - (optionally) patches for kazoo's setup.py so that it doesn't search for and try to download any dependencies. This can be done in a later ticket, though. I note that kazoo can be run without installation: you can untar it, cd into the directory, and if you run python from there, you can immediately "import kazoo" and use it. Hence we could avoid setup.py completely and just copy the "kazoo" subdirectory into some directory in the PYTHONPATH. | 2 |
| 208 | DM-612 | remove obsolete QMS-related code | try running {code} find core css admin client tests site_scons | xargs grep -i qms {code} There is a lot of old unused qms related code. | 2 |
| 209 | DM-613 | Automated test differences after CSS migration | I'm running automated tests with recent master after merging DM-605 and observe some differences between mysql and qserv. Here I'm going to document all differences (and everything related). | 6 |
| 210 | DM-614 | rename qserv_admin.py to qserv-admin.py | We have 6 scripts in qserv/admin/bin that start with "qserv-", and one that starts with "qserv_", we should rename qserv_admin to qserv-admin. | 1 |
| 211 | DM-616 | commons.config should always be managed as a global variable | Qserv configuration object (i.e. commons.config) used in admin and tests should be managed the same way a logger object is (cf. http://hg.python.org/cpython/file/2.7/Lib/logging/__init__.py). I.e. a function called config.getConfig("config-type") (returning an config occurence of a global dictionnary), should be used in order to ease retrieval of all configuration objects related to client, data, (and server?) anywhere in the admin/test python code. | 2 |
| 212 | DM-621 | User friendly single node loading script | This entails the creation of an end-to-end loading script that, given database and table param files, a SQL table schema and one or more CSV data files, places appropriate metadata into zookeeper, runs the partitioner, and finally creates and loads chunk tables. | 8 |
| 213 | DM-622 | Qserv configuration tool refactoring | * Instance configuration should not be in the same directory where qserv software is installed Qserv configuration, including things like mysql or zookeeper data directory should be separate from where Qserv software is installed. At the moment, when I want to work with branch "a" and "b", in order to switch from one to the other I have to run "cd $QSERV_DIR/admin; scons". This is ok the very first time after I created a branch, but it is way too heavy afterwards, because it wipes out all data from zookeeper and mysql. | 10 |
| 214 | DM-623 | Package antlr 2.7 in eups | NULL | 5 |
| 215 | DM-624 | Make sure we have queryId in all places where needed in logging | NULL | 4 |
| 216 | DM-625 | Too many connections from czar to zookeeper | I have just managed to crash qserv czar by running repeated queries against it. What I see in the logs: qserv-czar.log: {code} 2014-05-02 13:21:10,861:22525(0x7f76e37ab700):ZOO_ERROR@handle_socket_error_msg@1723: Socket [127.0.0.1:12181] zk retcode=-4, errno=104(Connection reset by peer): failed while receiving a server response 2014-05-02 13:21:10,861:22525(0x7f76e37ab700):ZOO_ERROR@handle_socket_error_msg@1723: Socket [127.0.0.1:12181] zk retcode=-4, errno=104(Connection reset by peer): failed while receiving a server response terminate called after throwing an instance of 'lsst::qserv::css::CssException_ConnFailure' what(): Failed to connect to persistent store. {code} and zookeeper.log: {code} 2014-05-02 13:21:10,861 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:12181:NIOServerCnxnFactory@193] - Too many connections from /127.0.0.1 - max is 60 2014-05-02 13:21:10,861 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:12181:NIOServerCnxnFactory@193] - Too many connections from /127.0.0.1 - max is 60 2014-05-02 13:21:14,585 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:12181:NIOServerCnxn@357] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145bdd1b8440015, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:744) 2014-05-02 13:21:14,585 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:12181:NIOServerCnxn@1007] - Closed socket connection for client /127.0.0.1:49913 which had 2014-05-02 13:21:14,585 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:12181:NIOServerCnxn@357] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145bdd1b8440016, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:744) 2014-05-02 13:21:14,586 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:12181:NIOServerCnxn@1007] - Closed socket connection for client /127.0.0.1:49939 which had {code} | 1 |
| 217 | DM-626 | ORDER BY and DISTINCT do not work reliably in qserv | Queries with ORDER BY and DISTINCT are buggy. For example, results do not always come ordered and order changes from one run to another: {code} mysql> SELECT objectId FROM Object WHERE qserv_areaspec_box(0, 0, 3, 10) ORDER BY objectId; +-----------------+ | objectId | +-----------------+ | 417857368235490 | | 417861663199589 | | 417865958163688 | | 420949744686724 | | 420954039650823 | | 424042121137958 | | 424046416102057 | | 427134497589192 | | 430226874040426 | +-----------------+ 9 rows in set (1.27 sec) mysql> SELECT objectId FROM Object WHERE qserv_areaspec_box(0, 0, 3, 10) ORDER BY objectId; +-----------------+ | objectId | +-----------------+ | 424042121137958 | | 424046416102057 | | 427134497589192 | | 430226874040426 | | 417857368235490 | | 417861663199589 | | 417865958163688 | | 420949744686724 | | 420954039650823 | +-----------------+ 9 rows in set (1.24 sec) {code} This was done with testdata/case01 data, let me know if you need to load that data. Also, using case03 data, e.g. {code} SELECT distinct run, field FROM Science_Ccd_Exposure WHERE run = 94 AND field = 535; {code} returns 6 rows in qserv (vs 1 in mysql). The full list of DISTINCT failures is (all with testdata/case03): - 0002_fetchRunAndFieldById.txt - 0021_selectScienceCCDExposure.txt - 0030_selectScienceCCDExposureByRunField.txt | 1 |
| 218 | DM-627 | Switch to using new partitioner, loader | Integrated tests procedure has to rely on new loader | 4 |
| 219 | DM-630 | Non-partitioned table query returns duplicated rows | Running automated test I noticed that a query on non-partitioned table returns multiple copies of the same row, one copy per chunk. Here is example: {code} mysql> SELECT offset, mjdRef, drift FROM LeapSeconds where offset = 10; +--------+--------+-------+ | offset | mjdRef | drift | +--------+--------+-------+ | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | | 10 | 41317 | 0 | +--------+--------+-------+ 13 rows in set (5.62 sec) {code} Czar log file shows that it correctly finds that table is non-chunked but sends query to each chunk anyway: {code} 20140502 16:22:08.745081 0x3172430 INF *** KvInterfaceImplZoo::exist(), key: /DATABASES/LSST/TABLES/LeapSeconds/partitioning 20140502 16:22:08.745735 0x3172430 INF *** LSST.LeapSeconds is NOT chunked. 20140502 16:22:08.745762 0x3172430 INF *** KvInterfaceImplZoo::get2(), key: /DATABASES/LSST/TABLES/LeapSeconds/partitioning/subChunks 20140502 16:22:08.746393 0x3172430 INF *** LSST.LeapSeconds is NOT subchunked. 20140502 16:22:08.746409 0x3172430 INF getChunkLevel returns 0 ..... 20140502 16:22:08.757832 0x3172430 INF <py> Using 85 stripes and 12 substripes. 20140502 16:22:08.775586 0x3172430 INF <py> Using /usr/local/home/salnikov/dm-613/build/dist/etc/emptyChunks.txt as default empty chunks file. 20140502 16:22:08.791559 0x3172430 INF <py> empty_LSST.txt not found while loading empty chunks file. 20140502 16:22:08.791592 0x3172430 ERR <py> Couldn't find empty_LSST.txt, using /usr/local/home/salnikov/dm-613/build/dist/etc/emptyChunks.txt. 20140502 16:22:08.891239 0x7fbb28003660 INF QuerySession::_buildChunkQueries() : Non-subchunked 20140502 16:22:08.891498 0x7fbb28003660 INF Msg cid=6630 with size=153 20140502 16:22:08.891682 0x7fbb28003660 INF Added query id=6630 url=xroot://qsmaster@127.0.0.1:1094//q/LSST/6630 with save /dev/shm/qserv-salnikov-b93a8b7cca1128f50fa5531feb93f8f24a185f162d36c10ee76b8dca/1_6630_0 20140502 16:22:08.891694 0x7fbb28003660 INF Opening xroot://qsmaster@127.0.0.1:1094//q/LSST/6630 20140502 16:22:08.891705 0x7fbb28003660 INF QuerySession::_buildChunkQueries() : Non-subchunked 20140502 16:22:08.891882 0x7fbb28003660 INF Msg cid=6631 with size=153 20140502 16:22:08.892077 0x7fbb28003660 INF Added query id=6631 url=xroot://qsmaster@127.0.0.1:1094//q/LSST/6631 with save /dev/shm/qserv-salnikov-b93a8b7cca1128f50fa5531feb93f8f24a185f162d36c10ee76b8dca/1_6631_0 20140502 16:22:08.892087 0x7fbb28003660 INF Opening xroot://qsmaster@127.0.0.1:1094//q/LSST/6631 20140502 16:22:08.892097 0x7fbb28003660 INF QuerySession::_buildChunkQueries() : Non-subchunked 20140502 16:22:08.892275 0x7fbb28003660 INF Msg cid=6800 with size=153 20140502 16:22:08.892462 0x7fbb28003660 INF Added query id=6800 url=xroot://qsmaster@127.0.0.1:1094//q/LSST/6800 with save /dev/shm/qserv-salnikov-b93a8b7cca1128f50fa5531feb93f8f24a185f162d36c10ee76b8dca/1_6800_0 ... {code} Looking at the code together with Daniel we found that at the Python level (czar/app.py) the code that dispatches query does not check for chunkLevel, this is likely why this happens. The code to look at is in {{InbandQueryAction._applyConstraints()}} method. | 5 |
| 220 | DM-631 | Define C++ API for C++ Geometry | NULL | 10 |
| 221 | DM-633 | Query sessions are never destroyed | Please see DM-625, when I run say 10 "select count(*) from LSST.Object" queries, for each query a new AsyncQueryManager is created in dispatcher, but the sessions are never destroyed. | 3 |
| 222 | DM-635 | admin/tests/test_qservAdminImpl.py has hardcoded connection info | NULL | 1 |
| 223 | DM-637 | complexity of eups dependencies relationships for db package | Hello, I'm currently trying to use the very last version of db package (the one which relies on sconsUtils), but, in order to make it works with Qserv, I had to introduce next update : {code:bash} fjammes@clrlsstwn02-vm:~/src/qserv-packager/dist/dependencies/db (master) $ git diff HEAD~1 diff --git a/ups/db.cfg b/ups/db.cfg index e1ae31b..a469061 100644 --- a/ups/db.cfg +++ b/ups/db.cfg @@ -3,7 +3,7 @@ import lsst.sconsUtils dependencies = { - "required": ["mysqlclient", ], + "required": ["mysql", ], } config = lsst.sconsUtils.Configuration( diff --git a/ups/db.table b/ups/db.table index 8c8d831..9e770a3 100644 --- a/ups/db.table +++ b/ups/db.table @@ -1,5 +1,5 @@ setupRequired(python) -setupRequired(mysqlclient) +setupRequired(mysql) setupRequired(mysqlpython) setupRequired(sconsUtils) {code} Is there a solution to describe in eups that mysqlclient is included in mysql ? Thanks, Fabrice | 1 |
| 224 | DM-644 | update overview docs to clarify roles of meas_multifit and shapelet packages | From the review of DM-17: {quote} It was not obvious how responsibility is split between {{meas_extensions_multishapelet}}, {{shapelet}}, and {{meas_multifit}}. Shapelet could use an {{overview.dox}} file. {quote} {{meas_extensions_multiShapelet}} is on its way out, so we'll wait until that's done and then document the relationship between meas_multifit and shapelet. | 2 |
| 225 | DM-645 | meas_base plugin for sampling-based galaxy fitter | In addition to a CModel plugin for galaxy photometry (DM-240), we should create a plugin to do sampling-based galaxy fitting, using the optimizer as a starting point and the existing AdaptiveImportanceSampler class to do most of the work. A major blocker for this is the fact that the {{SourceTable/SourceRecord}} don't currently provide any way to save multiple samples per object. This issue may get worked on before it falls into an official sprint, as it's something I want to on my 20% time. | 8 |
| 226 | DM-646 | Implement DISTINCT aggregate in qserv | It looks like DISTINCT aggregate is not supported yet in qserv. Daniel told me that this should be relatively straightforward to add. Adding this ticket so that we do not forget it. | 2 |
| 227 | DM-647 | support samples and other many-to-one outputs in SourceRecord | For DM-645, I need to be able to save multiple samples (each a parameter vector and possibly a weight) with each {{SourceRecord}}, with the number of samples possibly differing from {{SourceRecord}} to {{SourceRecord}}. I'll also want to save Mixture distributions, which can also be easily represented as a sequence-of-tuples of values to be associated with a record. Since we don't want to hard-code the outputs of a particular plugin into {{SourceRecord}} (which is what I've done with the {{ModelFitRecord}} class in meas_multifit, which I'd like to remove), I think this means we want a more general way of allowing algorithms to define many-to-one tables to be associated with each source. While few algorithms will likely want to save samples, this plugin may not be the only one, and it's likely a general many-to-one feature could be used by other algorithms to save diagnostics when configured to do so. I'm worried this is another case where afw::table may be intruding on a feature best left to a true RDBMS, but I don't like the idea of tying an RDBMS directly to the source measurement framework either. I'm open to other ideas, if anyone has one. It's also worth pointing out that because this is blocking DM-645, which is important for my 20%-time science work, I'd like to get moving on this even before it becomes a real priority for LSST. But I'd still like to get some design feedback on this. | 20 |
| 228 | DM-648 | Add support for running unit tests in scons | Add code in scons that runs unit tests for Qserv. | 5 |
| 229 | DM-649 | framework for documenting "how to run qserv" | We need to have permanent location for how-to-run-qserv, currently we keep it in https://dev.lsstcorp.org/trac/wiki/db/Qserv/RedesignFY2014/Hackathon2/howToRunQserv. It should be in the code repo. Need to decide on how we format it, and need to expose it on our Qserv trac/confluence pages | 3 |
| 230 | DM-650 | Migrate database trac pages to Confluence | Need to migrate Qserv trac pages to Confluence. | 10 |
| 231 | DM-653 | Run baseline HTCondor ClassAds Scenarios | Run initial HTCondor ClassAds Scenarios to verify that the implementation for utilizing Rank to place Jobs near data is operating as anticipated. The main test of the baseline scenarios is to verify that, e.g., a job for a CCD that has calibrations advertised for a particular slot/node will consistency be executed within that location for appropriate Rank expression. This is to occur even when other open slots are always available (e.g., 4 CCDs, 5 slots). | 4 |
| 232 | DM-654 | Run "single slow worker" HTCondor ClassAds Scenario | We run and study a "single slow worker" HTCondor ClassAds Scenario. The scenario is a perturbation of the baseline HTCondor ClassAds Scenario. In the baseline, Jobs for a given CCD are consistently pinned to a slot/node that advertises the presence of associated data files/calibration files for that CCD. A baseline run may proceed, for example, with jobs for 4 CCDs repeating executing within same HTCondor slot on a node (even when spare processing slots are readily available.). The baseline is observed to be quite stable, as the pool is empty each time a wave of jobs is submitted. In the "single slow worker" scenario, we cause one of the jobs for a chosen CCD to stall (mocking up a slow file transfer, lengthy computation in an algorithm, etc), such that the pool is not empty at the the submission time for a wave of jobs. We seek to observe how Rank places jobs in this scenario, and work to assign Rank (especially for spare slots) in an optimal way so as to minimize file transfers. | 4 |
| 233 | DM-655 | unknown column derails Qserv | Running a query that references invalid column hangs Qserv, it looks like the query is never squashed. For example, I run a query: {code} SELECT distinct run, field FROM Science_Ccd_Exposure WHERE run = 94 AND field = 535; {code} on the pt1.1 data set Corresponding log from xrootd: {code} Foreman:>>Unknown column 'run' in 'field list' Unable to execute query: CREATE TABLE r_7d8e5a45a6aba94de141cf1d8ab510c89_3598_0 SELECT run,field FROM LSST.Science_Ccd_Exposure \ AS QST_1_ WHERE run=94 AND field=535; <<---Error with piece 0 complete (size=1). Foreman:TIMING,q_f99cQueryExecFinish,1399667470 Foreman:Broken! ,q_f99cQueryExec---Unknown column 'run' in 'field list' Unable to execute query: CREATE TABLE r_7d8e5a45a6aba94de141cf1d8ab510c89_3598_0 SELECT run,field FROM L\ SST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535; (during QueryExec) QueryFragment: CREATE TABLE r_7d8e5a45a6aba94de141cf1d8ab510c89_3598_0 SELECT run,field FROM LSST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535; Foreman:Fail QueryExec phase for q_f99c: Unknown column 'run' in 'field list' Unable to execute query: CREATE TABLE r_7d8e5a45a6aba94de141cf1d8ab510c89_3598_0 SELECT run,field \ FROM LSST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535; (during QueryExec) QueryFragment: CREATE TABLE r_7d8e5a45a6aba94de141cf1d8ab510c89_3598_0 SELECT run,field FROM LSST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535; Foreman:(FinishFail:0x7f26700055c0) Db = q_f99cc9cd5519d465e673119a84b5570a, dump = /usr/local/home/becla/qserv/1/qserv/build/dist/xrootd-run/result/f99cc9cd5519d465e673119a84b\ 5570a hash=f99cc9cd5519d465e673119a84b5570a Foreman:Finished task Task: msg: session=7 chunk=3598 db=LSST entry time=Fri May 9 15:31:10 2014 frag: q=SELECT run,field FROM LSST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535, sc= rt=r_7d8e5a45a6aba94de141cf1d8ab510c89_3598_0 ScanSched:ChunkDisk remove for 3598 : SELECT run,field FROM LSST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535 BlendSched:Completed: (3598)SELECT run,field FROM LSST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535 ScanSched:Completed: (3598)SELECT run,field FROM LSST.Science_Ccd_Exposure AS QST_1_ WHERE run=94 AND field=535 ScanSched:_getNextTasks(32)>->-> ScanSched:ChunkDisk busyness: no ScanSched:_getNextTasks <<<<< BlendSched:Blend trying other sched. GroupSched:_getNextTasks(4)>->-> GroupSched:_getNextTasks <<<<< 140509 15:31:27 13960 cms_Finder: Waiting for cms path /usr/local/home/becla/qserv/1/qserv/build/dist/tmp/worker/.olb/olbd.admin {code} and the log in czar has {code} 20140509 15:35:40.070654 0x7f4694003660 INF Still 1 in flight. {code} | 4 |
| 234 | DM-658 | Database name used by integration tests should use their own dedicated database, not "LSST" | Currently automated tests use the database "LSST", and they will go ahead and delete that database without any warning. This is far from ideal, we should have a more unique database name. How about something like qservAutoTest_<uniqueId>. | 9 |
| 235 | DM-661 | Parser has inverted order for "limit" and "order by" | {code} SELECT run FROM LSST.Science_Ccd_Exposure order by field limit 2 {code} Works in MySQL, fails in Qserv (ERROR 4120 (Proxy): Error executing query using qserv.) {code} SELECT run FROM LSST.Science_Ccd_Exposure limit 2 order by field {code} Works in Qserv, fails in MySQL (limit should be after order by) | 1 |
| 236 | DM-664 | "out of range value" message when running qserv-testdata (loader.py) | Fabrice I am getting "out of range value" when I run the qserv-testdata: Are you seeing that too? 2014-05-09 18:11:55,975 {/usr/local/home/becla/qserv/1/qserv/build/dist/lib/python/lsst/qserv/admin/commons.py:134} INFO stderr : /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'calib_detected' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'calib_psf_candidate' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'calib_psf_used' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_negative' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_badcentroid' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'centroid_sdss_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_edge' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_interpolated_any' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_interpolated_center' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_saturated_any' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_saturated_center' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_cr_any' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_cr_center' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'centroid_gaussian_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'centroid_naive_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'shape_sdss_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'shape_sdss_centroid_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'shape_sdss_flags_unweightedbad' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'shape_sdss_flags_unweighted' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'shape_sdss_flags_shift' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'shape_sdss_flags_maxiter' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_psf_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_psf_flags_psffactor' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_psf_flags_badcorr' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_naive_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_gaussian_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_gaussian_flags_psffactor' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_gaussian_flags_badcorr' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flux_sinc_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_psf_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_psf_flags_maxiter' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_psf_flags_tinystep' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_psf_flags_constraint_r' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_psf_flags_constraint_q' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flux_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flags_psffactor' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flags_badcorr' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flags_maxiter' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flags_tinystep' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flags_constraint_r' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flags_constraint_q' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_dev_flags_largearea' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flux_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flags_psffactor' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flags_badcorr' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flags_maxiter' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flags_tinystep' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flags_constraint_r' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flags_constraint_q' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_exp_flags_largearea' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_combo_flux_flags' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_combo_flags_psffactor' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'multishapelet_combo_flags_badcorr' at row 1 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'calib_detected' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'calib_psf_candidate' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'calib_psf_used' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_negative' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_badcentroid' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'centroid_sdss_flags' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_edge' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_interpolated_any' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_interpolated_center' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_saturated_any' at row 2 self.cursor.execute(stmt) /usr/local/home/becla/qserv/1/qserv/build/dist/bin/loader.py:99: Warning: Out of range value for column 'flags_pixel_saturated_center' at row 2 self.cursor.execute(stmt) | 2 |
| 237 | DM-666 | partition package has to detect eups-related boost | partition package doesn't detect eups-related boost. This has to be fixed by using sconsUtils, or hand-made procedure. {code:bash} [fjammes@lsst-dev lsstsw]$ export LD_LIBRARY_PATH="$LSSTSW/anaconda/lib:$LD_LIBRARY_PATH" [fjammes@lsst-dev lsstsw]$ setup boost 1.55.0.1+1 [fjammes@lsst-dev lsstsw]$ rebuild partition partition: ok (0.5 sec). boost: ok (0.3 sec). python: ok (0.3 sec). scons: ok (0.4 sec). # BUILD ID: b49 python: master-gcbf93ab65b (already installed). scons: 2.1.0+8 (already installed). boost: 1.55.0.1+1 (already installed). partition: master-gf2ef2cf2dc ERROR (1 sec). *** error building product partition. *** exit code = 1 *** log is in /lsst/home/fjammes/src/lsstsw/build/partition/_build.log *** last few lines: ::::: scons: Reading SConscript files ... ::::: Checking for C++ library boost_system-mt... no ::::: Checking for C++ library boost_system... no ::::: Checking for C++ library boost_thread-mt... no ::::: Checking for C++ library boost_thread... no ::::: Checking for C++ library boost_filesystem-mt... no ::::: Checking for C++ library boost_filesystem... no ::::: Checking for C++ library boost_program_options-mt... no ::::: Checking for C++ library boost_program_options... no ::::: Missing required boost library! # BUILD b49 completed. {code} | 3 |
| 238 | DM-674 | fix handling of nested control objects | Work on the HSC side has revealed some problems with nested control objects being wrapped into config objects. This is a pull request for those changes (along with writing a unit test for some of them). Some (but not all of these changes) are part of Trac ticket #3163 (https://dev.lsstcorp.org/trac/ticket/3163), which I'll now close as a duplicate. | 2 |
| 239 | DM-675 | Citizen methods should be private and accessible only through a friend interface | The Citizen interface is useful, but it pollutes its derived classes with methods and attributes that can cause confusion later on (I've got a concrete example of that confusion that Perry and I just spent a few days tracking down - Citizen's {{getId()}} was being mistaken for {{SourceRecord.getId()}}). I think everything Citizen provides should be hidden and only accessible through a friend interface, e.g.: {code} afw::image::Image<float> image(4, 5); daf::base::CitizenAccess::getId(image); {code} We should also make an effort to ensure that other aspects of Citizen's design don't affect derived classes, perhaps by prefixing an name that could be seen by derived classes with a "Citizen" prefix; see https://dev.lsstcorp.org/trac/ticket/2461. | 4 |
| 240 | DM-676 | Implement HTCondor dynamic classad solution for Slot based values | The HTCondor team will be updating their HOWTO for managing Slot based classads/dynamic classads set by a cron startd process. We currently have a technique for dynamic slot based values that is iinefficient from a negotiation perspective, and we will want to update to a more optimal approach that the HTCondor team plans to provide. | 2 |
| 241 | DM-677 | Develop monitoring for identifying Data processed on a Node/in a Slot | To understand the effectiveness with which we are mapping Jobs to Data, it is vital to monitor/record what data has been processed on a given Node, or within a given Slot on a Node. Under this issue we examine HTCondor monitoring standards like STARTD_HISTORY, as well as more custom implementation of blackboard type records via Job Update hooks to be executed on the execute node (along the lines of OWL.) | 4 |
| 242 | DM-678 | Run HTCondor ClassAds Scenarios with heterogeneous data cache | The initial series of tests with HTCondor ClassAds work with jobs for individual ccds with a single file representing the data dependency for the job. In this issue we consider the management of multiple types of data dependencies that may have to be cached for jobs (calibrations, templates, catalogs of sources/objects, etc). | 4 |
| 243 | DM-680 | Study ORDER BY support | We don't have a proper implementation of ORDER BY. Actually, to support ORDER BY properly, we really have to manage all the column names, so we would need to have an in-memory list of columns for the table in question. This is because the general case requires us to ORDER BY a column that may not exist in the select list. In this case, we must add it to the select list if it is not there (or apply *). The easiest solution is to only allow ORDER BY if the sort key column exists explicitly in the select list. | 4 |
| 244 | DM-681 | Parser ignores syntax after LIMIT | Parser stops after the LIMIT condition, believing that it has a complete select statement. It ignores whatever is afterwards.To fix this, we would need to alter the parser to make sure that there isn't garbage afterwards. So we have to make sure that the only thing acceptable afterwards is a semicolon or a comment. The right thing to do is probably to add a grammar rule for this. The easiest thing is to check to see if the entire string was consumed (give-or-take a semicolon), but this would disallow comments. | 4 |
| 245 | DM-684 | Estimate expected counts of unassociated sources | We need to have an idea how many sources/forcedSource/diaSources that are not associated with any object we will have to deal with in the database. Can we have a rough estimate? (e.g., per DR). | 2 |
| 246 | DM-685 | Fine-tune logging messages | Fine-tune log messages in Qserv (what messages are printed, what is the error level, etc) | 5 |
| 247 | DM-689 | During scons configure : check if mysql isn't runing | Mysqld can't be configured is its running before configuration step. | 1 |
| 248 | DM-690 | Minor possible enhancements in install procedure | Usefull enhancements : - add swigged target to "build" alias (run scons install to see that swigged target are re-builded at install time) Other possibles enhancements : - manage default (i.e. const.py) for server configuration file ? - state.py : where to save state ? print it to sdtout ? | 7 |
| 249 | DM-693 | Create tutorial on the use of eups | Create a beginner's guide to the use of EUPS for the DM Developer Guide. Describe the capabilities and basic usage for "everyday" user and developer activities. Provide pointers to the EUPS reference manual for advanced usage. | 10 |
| 250 | DM-699 | rename git repository qservdata to qserv_testdata | eups package have the same name as their related git repos. Renaming git repos would lead to a more understandable name. Please note that the qserv-testdata may also be cloned from qservdata, and and qservdata be removed. New repos will also have to be distributed with lsst-sw tool. | 1 |
| 251 | DM-702 | Buildbot CI needs to save manifest file of failed build for later user debug | The manifest file created during a build instance is transient and removed as soon as the next build commences. Due to that volatility, it's important to save the manifest to some well-known location so that the developer responsible for debug and repair can easily setup the failing environment. The location of the manifest file will be provided to the developer(s) in the failure notification. | 4 |
| 252 | DM-703 | Use of HipChat for Buildbot CI failure notifications should be explored | K-T recommended the use of HipChat rather than email when notifying users of a buildbot build failure. The purpose was twofold: get immediate attention from the developers and help change the culture towards using HipChat more. This Issue is to explore the feasibility of using HipChat for the notifications. | 1 |
| 253 | DM-704 | Better review notification e-mails | Russell writes: {quote} I think our system for getting code reviewed using JIRA needs some improvements. It seems that people don't always know that they have been assigned to review a ticket. Also, even if I know I have been assigned to review a ticket, I find it hard to find on JIRA. More concretely, I would like to see these improvements: - Much clearer notification that one has been assigned as a reviewer. Presently the email is quite generic and easy to miss. In fact I find that most JIRA notifications are rather hard to read -- it's not always easy to see what has changed and thus why I should care. The signal to noise ratio is poor. - By default a user should see which issues they have been assigned as reviewer when they log into JIRA. (If there is a way to reconfigure the dashboard for this, I'd like to know about it, but it really should be the default). One way to fix this, of course, is to reassig the ticket when putting it into review, but we have good reasons to avoid that. -- Russell {quote} and I added: {quote} In fact, you don't know that the ticket has passed into review unless you scroll all the way to the bottom of the comment. If the comment associated with the change in status is long and you don't scroll all the way down, then you may not know that you were assigned to review. With Trac, the important information was at the top of the e-mail. {quote} | 2 |
| 254 | DM-705 | CSS keys are too fine-grain, consider merging them together (design) | I've seen a lot of sentiment (from Serge and Daniel) to try and combine keys in CSS. Current design has one key for each table-chunk (leading to tens of thousands of keys in production), and small-grain keys like "nStripes", "nSubStripes", and "overlap" in partitioning are each stored in separate key. This issue will capture discussion, decisions, and implementation of a more compact version. | 10 |
| 255 | DM-706 | cleanup extra file names in docstring | Reported by Serge in email: When using doxygen to document C++ source, you can mark a comment block with just: {code} /** @file * Blah blah */ {code} in which case doxygen assumes you want the comment block tied to the file it appears in. We seem to have lots of "@file <fileName>” statements all over the place, which is an extra thing we have to remember to change when renaming files. Is there some reason to do it that way that I’m missing? | 1 |
| 256 | DM-707 | cleanup exception code in CSS | Reported by Serge: In CssException.h you’ve got: {code} class CssRunTimeException: public std::runtime_error { … }; class CssException_XXXX : public CssRunTimeException { … }; {code} This is inconsistent (shouldn’t it be CssRunTimeException_XXX, or maybe even CssRunTimeError?), lengthy, violates the LSST C++ naming conventions, and doesn’t match the KvInterface docs, which all still talk about a CssException class that does not exist. Can we consider changing this to something more like: {code} class CssError : public std::runtime_error class KeyError : public CssError class NoSuchTable : public KeyError class NoSuchDb : public KeyError class AuthError : public CssError class ConnError : public CssError {code} ? Then we can succinctly throw and catch css::NoSuchTable, css::AuthError etc… | 2 |
| 257 | DM-708 | IN2P3: Cloud-computing Platform / OpenStack | CC-IN2P3 offer computing resources via Openstack : https://indico.in2p3.fr/getFile.py/access?sessionId=2&resId=0&materialId=0&confId=8236 This platform allow to boot virtual machines on multiples linux distribution. It could be a powerfull tool for continuous integration and testing. Virtual machines are easilly available to all IN2P3 users, only a SSH account on ccage.in2p3.fr is required. | 15 |
| 258 | DM-709 | Prepare a fedora64 openstack image which allow to easily build and test Qserv | Qserv packaging procedure requires to often rebuild Qserv and relaunch integration tests. IN2P3 Openstack platform offer next virtual machines : {code:bash} [fjammes@ccage030 ~]$ nova flavor-list {code} | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | | 1 | m1.tiny | 512 | 0 | 0 | | 1 | 1.0 | True | | 15 | cc.windows.small | 4096 | 20 | 0 | | 2 | 1.0 | True | | 16 | cc.windows.xlarge | 8192 | 50 | 0 | | 4 | 1.0 | True | | 2 | m1.small | 2048 | 10 | 20 | | 1 | 1.0 | True | | 3 | m1.medium | 4096 | 10 | 40 | | 2 | 1.0 | True | | 4 | m1.large | 8192 | 10 | 80 | | 4 | 1.0 | True | | 5 | m1.xlarge | 16384 | 10 | 160 | | 8 | 1.0 | True | | 6 | cc.lsst.medium | 4096 | 20 | 40 | | 2 | 1.0 | False | | 7 | cc.lsst.large | 16384 | 20 | 160 | | 8 | 1.0 | False | | 9 | cc.lsst.xlarge | 40000 | 20 | 160 | | 20 | 1.0 | False | cc.lsst.xlarge would allow a quick build/test of new Qserv release. | 5 |
| 259 | DM-710 | Reduce and comment client configuration file | Client configuration file '~/.lsst/qserv.conf) is used by integration test procedure. Next improvments are required : 1. use templates in it 2. client config file should retrieve templated values from mother config file" | 1 |
| 260 | DM-720 | Upgrade various external packages | I note that the LSST and HSC versions of external packages are slightly out of sync. I propose uprevving the LSST packages to match as HSC has tested these versions. {quote} cfitsio 3.360 HSC cfitsio 3310+2 sims Winter2014 current b4 b5 b6 b3 doxygen 1.8.2+2 sims b4 Winter2014 current b5 b6 b3 doxygen 1.8.5 HSC eigen 3.1.1+2 Winter2014 current b5 b6 b3 b4 eigen 3.2 HSC fftw 3.3.2+2 Winter2014 current b5 b6 b3 b4 fftw 3.3.3 HSC gsl 1.15+2 Winter2014 current b5 b6 b3 b4 gsl 1.16 HSC minuit2 5.22.00+2 Winter2014 current b5 b6 b3 b4 minuit2 5.28.00 HSC mysqlclient 5.1.65+3 Winter2014 current b5 b6 b3 b4 mysqlclient 5.1.73 HSC pyfits 3.1.2+2 sims b4 Winter2014 current b5 b6 b3 pyfits 3.2 HSC scons 2.1.0+7 sims b4 Winter2014 current b5 b6 b3 scons 2.3.0 HSC sqlite 3.7.14+2 Winter2014 current b5 b6 b3 b4 sqlite 3.8.2 HSC wcslib 4.14 wcslib 4.14+3 b4 Winter2014 current b5 b6 b3 xpa 2.1.14+2 Winter2014 current b5 b6 b3 b4 xpa 2.1.15 HSC {quote} | 20 |
| 261 | DM-736 | Rewrite secondary index system and merge empty chunks functionality | We'd like to rewrite the indexing system so that it doesn't depend on a narrow innodb table. We'd also like to fold in the emptychunks functionality. secondary index: dirColumn->(chunkId, subChunkId) emptychunks: hasChunk = chunkID -> bool danielw has a dirt simple implementation of the "create" portion of the secondary index. | 30 |
| 262 | DM-737 | Rendering an IR node tree should produce properly parenthesized output | It appears that rendering a tree of IR nodes doesn't always result in correct generation of parentheses. Consider the following tree: {panel} * OrTerm ** BoolFactor *** NullPredicate **** ValueExpr ***** ValueFactor: ColumnRef("refObjectId") ** BoolFactor *** CompPredicate **** ValueExpr ***** ValueFactor: ColumnRef("flags") **** Token("<>") **** ValueExpr ***** ValueFactor: Const("2") {panel} which corresponds to the SQL for: "refObjectId IS NULL OR flags<>2". If one prepends this (via {{WhereClause.prependAndTerm()}}) to the {{WhereClause}} obtained by parsing "... WHERE foo!=bar AND baz<3.14159;" and renders the result using {{QueryTemplate}}, one obtains: {{... WHERE refObjectId IS NULL OR flags<>2 AND foo!=bar AND baz<3.14159}} This is equivalent to {{... WHERE refObjectId IS NULL OR (flags<>2 AND foo!=bar AND baz<3.14159)}} which doesn't match the parse tree - one should obtain: {{... WHERE (refObjectId IS NULL OR flags<>2) AND foo!=bar AND baz<3.14159}} This issue involves surveying all IR node classes and making sure that they render parentheses properly. {color:gray}(One way we might test for this is to parse queries containing parenthesized expressions where removal of the parentheses changes the meaning of the query. This would give us some IR that we can render to a string and reparse back into IR. If the rendering logic is correct, one should obtain identical IR trees).{color} Other possibilities that might explain the behavior above is that the input tree is somehow invalid or that {{WhereClause.prependAndTerm}} creates invalid IR. | 8 |
| 263 | DM-740 | Implement abstract base class for approximated or interpolated fields | The user of an Approximate or Interpolate object doesn't care which of these they have, once the object has been constructed, and we should make these inherit from a common base class that only contains an interface for accessing the interpolated/approximated function while making no assumptions about its functional form. The new class will represent a scalar field defined over an integer bounding box, and will have methods for evaluating the field at a point and creating an image of the field. We could also consider giving it arithmetic interoperability with Image. I don't have a strong candidate for the name; I've called it "BoundedField" on the HSC side but would like that to be revisited. Adding derived classes to replace the functionality currently in the Approximate and Interpolate classes will be on a separate issue. The main work on this issue is tweaking the design on the HSC side and getting signoff on the final design from LSST developers; the actual coding should be minimal, as it's just an interface and we already have a good starting point for it. The HSC-side issue (which also includes work that is part of DM-1124) is here: https://hsc-jira.astro.princeton.edu/jira/browse/HSC-796 and the associated git commits are here: https://github.com/HyperSuprime-Cam/afw/compare/releases/S14A_0...tickets/DM-796 | 10 |
| 264 | DM-742 | Use geom eups package for installing geometry | Use geom eups package instead of downloading geometry.py during Qserv configuration step. | 3 |
| 265 | DM-744 | Qserv release (12.04) - final build, testing and cutting release | NULL | 4 |
| 266 | DM-746 | Simplify (script) install procedure | Install procedure described in READMEs.txt is complex and error prone. It could be encapsulated in two scripts : 1. the first for installing Qserv current version in the eups stack, following the LSST official install procedure, 2. the second, developer-oriented, for installing Qserv from a git repository to the eups stack installed during 1. This two scripts logs could be colorized for better ergonomy. | 3 |
| 267 | DM-750 | Integrate sciSQL in eups | In order to become compliant with eups, sciSQL install process may have to be refactored : - provide a scons build and install target which build and install sciSQL binary (.i.e. reduce waf tool features) - provide a configuration procedure (scisql-deploy.py) which install UDF in MySQL datadir, and deploy .so file in MySQL plugin directory. Then it has to be packaged in eups format. | 9 |
| 268 | DM-751 | Replacing boost system lib with eups libs breaks scons build | While detecting boost, Qserv build system checks for both system lib and then eups lib. This procedure use next code : {code:python} class BoostChecker: def __init__(self, env): self.env = env self.suffix = None self.suffixes = ["-gcc41-mt", "-gcc34-mt", "-mt", ""] self.cache = {} pass def getLibName(self, libName): if libName in self.cache: return self.cache[libName] r = self._getLibName(libName) self.cache[libName] = r return r def _getLibName(self, libName): state.log.debug("BoostChecker._getLibName() LIBPATH : %s, CPPPATH : %s" % (self.env["LIBPATH"], self.env["CPPPATH"])) if self.suffix == None: conf = self.env.Configure() def checkSuffix(sfx): return conf.CheckLib(libName + sfx, language="C++", autoadd=0) {code} and this last line run next gcc command : {code:bash} g++ -o .sconf_temp/conftest_10.o -c -g -pedantic -Wall -Wno-long-long -D_FILE_OFFSET_BITS=64 -fPIC -I/data/fjammes/stack/Linux64/protobuf/master-g832d498170/include -I/data/fjammes/stack/Linux64/boost/1.55.0.1/include -I/data/fjammes/stack/Linux64/zookeeper/master-gc48457902f/c-binding/include -I/data/fjammes/stack/Linux64/mysql/master-g5d79af2a50/include -I/data/fjammes/stack/Linux64/antlr/master-gc05368a54f/include -I/data/fjammes/stack/Linux64/xrootd/master-gfc9bfb2059/include/xrootd -Ibuild -I/data/fjammes/stack/Linux64/anaconda/1.8.0/include/python2.7 .sconf_temp/conftest_10.cpp g++ -o .sconf_temp/conftest_10 .sconf_temp/conftest_10.o -L/data/fjammes/stack/Linux64/xrootd/master-gfc9bfb2059/lib -L/data/fjammes/stack/Linux64/protobuf/master-g832d498170/lib -L/data/fjammes/stack/Linux64/antlr/master-gc05368a54f/lib -L/data/fjammes/stack/Linux64/zookeeper/master-gc48457902f/c-binding/lib -L/data/fjammes/stack/Linux64/mysql/master-g5d79af2a50/lib -L/data/fjammes/stack/Linux64/boost/1.55.0.1/lib -lboost_regex-mt scons: Configure: yes {code} As the "-mt" suffix is searched before the empty suffix, previous command succeed.In my example boost_regex-mt is a system lib. When launching "scons build", then CheckLib only looks for boost in /data/fjammes/stack/Linux64/boost/1.55.0.1/lib, not in /usr/lib/. This behaviour is eups-correct, but prevents to find boost_regex-mt. In this example, a trivial solution is to reverse self.suffixes in python code, but a better solution would be to prevent g++ to use default search paths (e.g. : /usr/lib and /usr/include) in the second command. Is it possible to to it with scons ? Mario, did you meet the same problem with sconsUtils ? Thanks Fabrice | 3 |
| 269 | DM-754 | Update obs_decam for new CameraGeom | The obs_decam package worked on by Paul and Andy B. needs to be updated to reflect changes in the camera geometry. | 6 |
| 270 | DM-758 | Stretch: Data and test script specification for daily/automated QA/integration tests | [I have renamed this Epic to better capture the current planned scope] Per discussions I am pulling this Epic into 02C.01.02 ["my" WBS] The intent for it is to cover the science and scientific programming portion activities associated with setting up a daily automated QA/integration run. The infrastructure activities to enable this are covered by other Epics. It is a stretch because it will be primarily worked on by the Tucson Scientist, still to be identified ICW Tucson Scientific Programmer, still to be identified. JK: In PMCS this would be New Hire LS6 | 70 |
| 271 | DM-760 | Identify Data Set for MiniProduction | NULL | 3 |
| 272 | DM-761 | Generate data for MiniProduction | Assuming that the data needed for mini-production are to be simulated, the input files need to be created and simulated. The input data then need to be put in a repo with appropriate calibrations. | 5 |
| 273 | DM-762 | Create scripts to run MiniProduction | Write command line script to run mini production. The scripts should be able to be handed directly to the orca layer. | 3 |
| 274 | DM-763 | Eliminate local pixel indexing; always use parent instead | As Jim Bosch notes in <https://dev.lsstcorp.org/trac/wiki/Winter2014/Bosch/Miscellaneous>: Our image classes currently handle two different pixel coordinate systems which differ only by the offset commonly referred to as "xy0". The PARENT coordinate system puts the first pixel in the image at xy0; the LOCAL coordinate system always puts the first pixel at (0,0). Our general rule has long been "always pay attention to xy0", which implies PARENT, but the Image class itself doesn't: in many operations, including subimage accessors and bbox getters, LOCAL is the default, whereas in others - particularly the pixel and iterator accessors - there isn't even an option to use PARENT. IMO this is the main source of image-offset bugs in the code. We plan to address this issue as follows: * Modify the methods on image-like objects that return pixel iterators or locators based x and/or y index, so that they use parent coordinates. * Eliminate the ImageOrigin enum argument, which appears in image-like constructors, image-like getBBox methods. * Eliminate the image origin string argument in the butler's get and put methods for image-like objects. At the end of this transition all indices will be PARENT indices; the concept of LOCAL will be gone. This will reduce ambiguity and eliminate an argument that is a source of confusion. Furthermore, at Jim's suggestion and K-T's agreement, we plan to do this in two basic stages: * Rename methods or otherwise make changes that should break any code using LOCAL indices. Fix the code accordingly. * Rename methods back and eliminate the ImageOrigin argument. Fix the code accordingly. | 53 |
| 275 | DM-764 | Exception naming convention | The naming convention for exceptions in pex_exceptions is quite redundant. This issue will make the convention more compact and update all packages that make use of pex_exceptions. | 5 |
| 276 | DM-765 | Evaluate moving to C++11 for .cc files | Check that {{C\+\+11}} works on .cc files. Make {{C\+\+11}} the default in SconsUtils. | 5 |
| 277 | DM-766 | Improve afw::CameraGeom::utils code | Some of the utility code in CameraGeom was not completely ported in W13 and documentation is in need of updating. | 3 |
| 278 | DM-767 | Determine scope of XY0 convention update | It's unclear exactly how much effort will be involved in making a change to how the XY0 is used. If the parent/child argument is removed completely this change could be quite invasive and wide reaching. | 2 |
| 279 | DM-768 | Does SWIG 3.x work with DM stack? | This task is to evaluate whether we can use SWIG 3.x (and is it stable enough). Assuming SWIG 3.x can be used with the stack, evaluate how much work will be involved in moving over to use it by default. | 10 |
| 280 | DM-769 | Create scripts to assess and report MiniProduction quality. | Write a command line script to assess the quality of the mini production run. This will involve comparing output data to data produced using a standard stack. The script will provide a report. | 7 |
| 281 | DM-770 | Create script to clean up after a MiniProduction run | The run of a mini production will produce an output repository. It's likely that we will not want to save all output data. A script to clean up and potentially save parts of the repo is needed. | 3 |
| 282 | DM-772 | Package log4cxx | Fabrice, can you package log4cxx? I should have asked you earlier, sorry I waited so long, not it becoming urgent! Bill is almost done with his logging prototype and will be turning it into a real package, and we need to have log4cxx packages. Many thanks. log4cxx version 0.10.0, which was released in 4/3/2008 but is still undergoing "incubation" at Apache. | 2 |
| 283 | DM-774 | XLDB in Rio | Prepare for and attend XLDB-South America in Rio. | 15 |
| 284 | DM-775 | XLDB-2015 report | Writing the report, most work done by Daniel, with input from Jacek and K-T. | 8 |
| 285 | DM-776 | Researching partnership opportunities | NULL | 8 |
| 286 | DM-778 | Restructure and package logging prototype | Restructure and package log4cxx-based prototype (currently in branch u/bchick/protolog). It should go into package called "log" | 8 |
| 287 | DM-780 | Access patterns for data store that supports data distribution | Data distribution related data store includes things like. chunk --> node mapping, locations of chunk replicas, runtime information about nodes (and maybe also node configuration?). Need to understand access patterns - who needs to access, how frequently etc. | 5 |
| 288 | DM-781 | research mysql cluster ndb | Checkout mysql cluster ndb from the perspective of data distribution - could it be potentially useful to store data related to data distribution? | 2 |
| 289 | DM-782 | Automated test should optionally ignore column headers | Some types of queries (like COUNT(*)) may return different column headers in qserv and mysql. This differences break our automated tests which dump and compare complete result including headers. It looks like we will not be able to guarantee that qserv can be made to return the same column headers as mysql except for providing aliases in the query itself. Running those queries without aliases is a legitimate use case so it would be nice to have an option in the test runner which ignores headers for some queries. | 4 |
| 290 | DM-783 | Disable failing test cases in automated tests | There are currently 4 test cases failing in out automated tests. Until we have a fix we want to disable them. | 1 |
| 291 | DM-786 | JOIN queries are broken | Running a simple query that does a join: {code} SELECT s.ra, s.decl, o.raRange, o.declRange FROM Object o JOIN Source s USING (objectId) WHERE o.objectId = 390034570102582 AND o.latestObsTime = s.taiMidPoint; {code} results in czar crashing with: {code} 2terminate called after throwing an instance of 'std::logic_error' what(): Attempted subchunk spec list without subchunks. {code} This query has been taken from integration tests (case01, 0003_selectMetadataForOneGalaxy.sql) | 3 |
| 292 | DM-787 | Understand DCR amplitudes using realistic distribution of stars | NULL | 4 |
| 293 | DM-788 | Determine refraction amplitudes as a function of SED | NULL | 2 |
| 294 | DM-789 | Model refraction amplitudes as a function of SED | NULL | 2 |
| 295 | DM-790 | Determine DCR ampltiudes as a function of SED | NULL | 2 |
| 296 | DM-791 | Model DCR amplitudes as a function of SED | NULL | 2 |
| 297 | DM-792 | Determine requirements on atmospheric measurements to predict refraction/DCR | NULL | 2 |
| 298 | DM-793 | Put together requirements to model refraction/DCR for a given source | NULL | 8 |
| 299 | DM-794 | SQL injection in czar/proxy.py | Running automated tests for some queries I observe python exceptions in czar log which look like this: {code} 20140529 19:47:19.364371 0x7faacc003550 INF <py> Query dispatch (7) toUnhandled exception in thread started by <function waitAndUnlock at 0x18cd8c0> Traceback (most recent call last): File "/usr/local/home/salnikov/qserv-master/build/dist/lib/python/lsst/qserv/czar/proxy.py", line 78, in waitAndUnlock lock.unlock() File "/usr/local/home/salnikov/qserv-master/build/dist/lib/python/lsst/qserv/czar/proxy.py", line 65, in unlock self._saveQueryMessages() File "/usr/local/home/salnikov/qserv-master/build/dist/lib/python/lsst/qserv/czar/proxy.py", line 87, in _saveQueryMessages self.db.applySql(Lock.writeTmpl % (self._tableName, chunkId, code, msg, timestamp)) File "/usr/local/home/salnikov/qserv-master/build/dist/lib/python/lsst/qserv/czar/db.py", line 95, in applySql c.execute(sql) File "/u2/salnikov/STACK/Linux64/mysqlpython/1.2.3+8/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/cursors.py", line 174, in execute self.errorhandler(self, exc, value) File "/u2/salnikov/STACK/Linux64/mysqlpython/1.2.3+8/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue _mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'r' AND sce.tract=0 AND sce.patch='159,3';', 1401410839.000000)' at line 1") ok 0.000532 seconds {code} I believe this is due to how query string is being constructed in czar/proxy.py: {code:py} class Lock: writeTmpl = "INSERT INTO %s VALUES (%d, %d, '%s', %f);" # ................... self.db.applySql(Lock.writeTmpl % (self._tableName, chunkId, code, msg, timestamp)) {code} If {{msg}} happens to contain quotes then resulting query is broken. One should not use Python formatting to construct query strings, instead the parameters should be passed directly to {{cursor.execute()}} method. | 2 |
| 300 | DM-800 | Zookeeper times out | I noticed running some queries, leaving system up and them returning few hours later and running more queries can result in: {code} ZOO_ERROR@handle_socket_error_msg@1723: Socket [127.0.0.1:12181] zk retcode=-4, errno=112(Host is down): failed while receiving a server response {code} It needs to be investigated (if we can reproduce) | 3 |
| 301 | DM-803 | Provide a "stackfitCalib" for coadds | Coadds have a Psf class that returns the proper sum of input PSFs at a point. We need the same functionality for the Calib object associated with the Coadd Exposure. This will require making a Calib a baseclass (it currently doesn't have a virtual dtor (although it does have virtual protected members)) | 4 |
| 302 | DM-813 | PhotoCalTask doesn't return information about which stars were used in calibration | The PhotoCalTask returns numpy arrays of the source and reference fluxes (and errors) of matched "good" photometric objects, typically stars. However, while estimating the zero point, it clips outliers so the actual list of objects used is shorter. Please add another output to the returned struct, a numpy Bool array "good", to indicate which objects are kept. | 1 |
| 303 | DM-814 | Cleanup in core/examples and core/doc | - core/examples and core/doc seems to be out of data. Some cleanup here would be welcome. | 1 |
| 304 | DM-815 | Define filter for Epics and mapping of Epics to WBS | NULL | 2 |
| 305 | DM-817 | qserv have to use boost from stack | To quote Jacek and KT: {code} Andy, re dm-751, KT says never use the system version. J. {code} So we need to switch qserv to eups-boost. This should be easy once DM-751 is done, just add boost to qserv.table. Then one can remove conditional part of {{BoostChecker}} which works with system-installed boost. | 1 |
| 306 | DM-820 | Simplify copying tables while adding columns | Currently, if I want to copy a table while adding a few columns (as specified by schema in the example) I need to do something like: {code} cat = afwTable.SourceCatalog(schema) cat.table.defineCentroid(srcCat.table.getCentroidDefinition()) cat.table.definePsfFlux(srcCat.table.getPsfFluxDefinition()) # etc. scm = afwTable.SchemaMapper(srcCat.getSchema(), schema) for schEl in srcCat.getSchema(): scm.addMapping(schEl.getKey(), True) cat.extend(srcCat, True, scm) {code} Please make this easier! For example - by adding a flag to the SchemaMapper constructor that automatically does the addMapping (should this be the default?) - by making it possible to copy all the slots (maybe this'll be the case when the new alias scheme is implemented?). Maybe we just need a new method: {code} cat = srcCat.extend(schema) {code} that does all the above steps. | 4 |
| 307 | DM-827 | Reimplement C++/Python Exception Translation | I'd like to reimplement our Swig bindings for C++ exceptions to replace the "LsstCppException" class with a more user-friendly mechanism. We'd have a Python exception hierarchy that mirrors the C++ hierarchy (generated automatically with the help of a few Swig macros). These wrapped exceptions could be thrown in Python as if they were pure-Python exceptions, and could be caught in Python in the same language regardless of where they were thrown. We're doing this as part of a "Measurement" sprint because we'd like to define custom exceptions for different kinds of common measurement errors, and we want to be able to raise those exceptions in either language. | 8 |
| 308 | DM-828 | Design Prototypes for C++ Algorithm API | We've never really been happy with the new design for the C++ algorithm API, and Perry and Jim have a few ideas to fix this that need to be fleshed out. Each of the subtasks of this issue will correspond to a different design idea. Ideally, for each one, we'll try to do a nearly-complete conversion of the SdssShape algorithm (as a good example of a complicated algorithm) to see how these ideas work in practice. | 6 |
| 309 | DM-829 | Algorithm API without (or with optional) Result objects | In this design prototype, I'll see how much simpler things could be made by making the main algorithm interface one that sets record values directly, instead of going through an intermediate Result object. Ideally the Result objects would still be an option, but they may not be standardized or reusable. | 3 |
| 310 | DM-832 | add persistable class for aperture corrections | We need to create a persistable, map-like container class to hold aperture corrections, with each element of the container being an instance of the class to be added in DM-740. A prototype has been developed on DM-797 on the HSC side: https://hsc-jira.astro.princeton.edu/jira/browse/HSC-797 and the corresponding code can be found on these changesets: https://github.com/HyperSuprime-Cam/afw/compare/32d7a8e7b75da6f5327fee65515ee59a5b09f6c7...tickets/DM-797 | 2 |
| 311 | DM-833 | implement coaddition for aperture corrections | We need to be able to coadd aperture corrections in much the same way we coadd PSFs. See the HSC-side HSC-798 and HSC-897 implementation for a prototype: https://hsc-jira.astro.princeton.edu/jira/browse/HSC-798 https://hsc-jira.astro.princeton.edu/jira/browse/HSC-897 with code here: https://github.com/HyperSuprime-Cam/meas_algorithms/compare/d2782da175c...u/jbosch/DM-798 https://github.com/HyperSuprime-Cam/meas_algorithms/compare/c4fcab3251...u/price/HSC-897a https://github.com/HyperSuprime-Cam/pipe_tasks/compare/6eb48e90be12d...u/price/HSC-897a | 3 |
| 312 | DM-834 | reduce code- and object-duplication in aperture correction and PSF coaddition | The aperture correction code to be added on DM-833 will likely not be as closely integrated with CoaddPsf as it could be, because the original design of CoaddPsf didn't anticipate the addition of other, similar classes. We should work on allowing these classes to share code. | 8 |
| 313 | DM-836 | design Array fields for table version 1 | While we're trying to eliminate the need for compound fields in afw::table, Arrays present a few problems. We could use FunctorKeys, the way we plan to use other compound fields, but here we need to guarantee that the per-element keys are contiguous, and we also be able to support views. We also need to determine the naming scheme. Finally, we need to make sure these work with aliases and slots. | 6 |
| 314 | DM-837 | Rewrite multiple-aperture photometry class | We've never figured out how to handle wrapping multiple-aperture photometry algorithms. They can't use the existing Result objects - at least not out of the box. We should try to write a new multiple-aperture photometry algorithm from the ground up, using the old ones on the HSC branch as a guide, but not trying to transfer the old code over. The new one should: - Have the option of using elliptical apertures (as defined by the shape slot) or circular apertures. - Have a transition radius at which we switch from the sinc photometry algorithm to the naive algorithm (for performance reasons). | 2 |
| 315 | DM-839 | Rename methods that return pixel iterators and locators in image-like classes, and change to use parent indexing | Image-like classes have a large number of methods that return pixel iterators and locators based on row and/or column. We wish to change these to use parent indexing (making row and column relative to XY0). As the first step in doing this, rename all these methods and modify them to parent indexing. Renaming will help identify and fix all code that uses these methods. | 6 |
| 316 | DM-840 | Change code so ImageOrigin must be specified (temporary) | Image-like classes have a getBBox method and various constructors that use an ImageOrigin argument which in most or all cases defaults to LOCAL. As the first stage in cleaning this up, try to break code that uses the default as follows: * Remove the default from getBBox(ImageOrigin) so an origin must be specified. * Change the default origin of constructors to a temporary new value UNDEFINED * Modify code that uses image origin to fail if origin is needed (it is ignored if bbox is empty) and is UNDEFINED. Note: this is less safe than changing constructors to not have a default value for origin, because the error will be caught at runtime rather than compile time. However, that is messy because then the bounding box will also have to be always specified, and possibly an HDU, so it would be a much more intrusive change. | 2 |
| 317 | DM-841 | Change data butler I/O of image-like objects to require imageOrigin if bbox specified (temporary) | As part of making PARENT the default for image origin, change the data butler to require that imageOrigin be specified if bbox is specified when reading or writing image-like objects. Note: this ticket turns out to be unnecessary, as all the few necessary change are done as part of DM-840. | 2 |
| 318 | DM-842 | Update code that uses pixel iterators and accessors | Fix all code that uses pixel iterators and locators as per the changes in DM-839. This affects roughly 60 files distributed over many packages. | 16 |
| 319 | DM-843 | Restore names of methods that return pixel iterators and locators | Restore the names of methods that return pixel iterators and pixel locators on image-like classes. (This is part of the final stage of eliminating LOCAL pixel indexing). | 2 |
| 320 | DM-844 | Eliminate ImageOrigin argument | Eliminate the ImageOrigin enum and argument from image-like classes. | 2 |
| 321 | DM-845 | Eliminate image origin argument from butler for (un)persisting image-like objects | Eliminate the image origin argument for butler get and put when dealing with image-like objects. | 2 |
| 322 | DM-846 | Change code to always specify image origin (temporary) | The next step after DM-840 is to change all code that uses image-like getBBox and constructors that take an image origin argument to always specify the origin (DM-840 will break any code that tries to use a default image origin). This results in a working stack after DM-840. Neither this nor DM-840 will be merged to master, though it can be used by HSC and other users as they transition to a default image origin of PARENT. See DM-1176 for the final step. | 10 |
| 323 | DM-847 | Update code to use restored names for methods that return pixel iterators and locators | For all code changed in DM-842, change it again to use the restored names for methods that return pixel iterators and locators. This task is much simpler than DM-842 because it is merely renaming methods. Nonetheless, it touches many files in many packages. | 4 |
| 324 | DM-848 | Specify ImageOrigin = PARENT in all cases | For all code that uses the ImageOrigin argument (either explicitly or using a bbox and the default value) change the code to use an explicit ImageOrigin of PARENT. In other words, update all code to match the changes in DM-840 and DM-841. This affects 40-ish modules in many packages. I am concerned about testing the modified code because we are missing unit tests for so many tasks. But we can do some data processing and check those results. I will want some help with that. | 12 |
| 325 | DM-849 | Eliminate use of ImageOrigin argument | For all code that uses the ImageOrigin argument, eliminate the use of that argument. In other words, update all code to match the changes in DM-844 and DM-845, This affects all the code affected by DM-848, plus any code that was already using ImageOrigin=PARENT, but is a simpler change. | 5 |
| 326 | DM-854 | duplicate column name when running near neighbor query | Running a simplified version of near neighbor query on test data from case01: {code} SELECT DISTINCT o1.objectId, o2.objectId FROM Object o1, Object o2 WHERE scisql_angSep(o1.ra_PS, o1.decl_PS, o2.ra_PS, o2.decl_PS) < 1 AND o1.objectId <> o2.objectId {code} Result in an error on the worker: {code} Foreman:Broken! ,q_38f9QueryExec---Duplicate column name 'objectId' Unable to execute query: CREATE TABLE r_13237cd4cfc9e0fa01497bcf\ 67a91add2_6630_0 SELECT o1.objectId,o2.objectId FROM Subchunks_LSST_6630.Object_6630_0 AS o1,Subchunks_LSST_6630.Object_6630_0 AS o2\ WHERE scisql_angSep(o1.ra_PS,o1.decl_PS,o2.ra_PS,o2.decl_PS)<1 AND o1.objectId<>o2.objectId; {code} It is fairly obvious what is going on. "SELECT t1.x, t2.x" is perfectly valid, but if we add "INSERT INTO SELECT t1.x, t2.x", we need to add names, eg. something like "INSERT INTO SELECT t1.x as x1, t2.x as x2" | 8 |
| 327 | DM-860 | Passing error messages from czar to end user - design | NULL | 3 |
| 328 | DM-861 | Provide a detailed integration tests report | Test output is very low level. Indeed only a verbose logfile and SLQ queries output are currently available. Furthermore failing queries (i.e. .sql.FIXME) aren't launched. This ouput could be leveraged to a detailed html web report (using sbadmin for example : http://startbootstrap.com/templates/sb-admin/) which could : - launch all queries, even failing ones, - print Qserv services status after each query, - execution time for each query, - information about MySQL and Qserv results differences, - href to Qserv services log files - all other interesting information about queries execution | 10 |
| 329 | DM-862 | apr and apt_util packages do not install shared library | When we installed apr and apr_utils packages (as a dependency of new log4cxx package, see DM-772) we discovered that both these packages only build static libraries but no shared libs are installed. This is problematic if mixed with shared libs and we use shared libs everywhere else. We certainly need to build shared libs for these packages, this ticket is to follow up on this problem. | 2 |
| 330 | DM-863 | near neighbor does not return results | A query from qserv_testdata (case01/queries/1051_nn.sql) runs through Qserv, but it returns no results, while the same query run on myql does return results. The exact query for qserv is: {code} SELECT o1.objectId AS objId FROM Object o1, Object o2 WHERE qserv_areaspec_box(0, 0, 0.2, 1) AND scisql_angSep(o1.ra_PS, o1.decl_PS, o2.ra_PS, o2.decl_PS) < 1 AND o1.objectId <> o2.objectId; {code} | 1 |
| 331 | DM-869 | disable extraneous warnings from boost (gcc 4.8) | Compiling qserv on ubuntu 14.04 (comes with gcc 4.8.2) results in huge number of warnings coming from boost. We should use the flag "-Wno-unused-local-typedefs". | 1 |
| 332 | DM-870 | XLDB Rio - workshop report | NULL | 3 |
| 333 | DM-873 | XLDB - strategic positioning | Discussions with strategic partners. Improving website and adding new context (community, speakers). 1-pager document | 3 |
| 334 | DM-874 | W'14 newinstall.sh picks up wrong python? | newinstall.sh fails with: Installing the basic environment ... Traceback (most recent call last): File "/tmp/test_lsst/eups/bin/eups_impl.py", line 11, in ? import eups.cmd File "/tmp/test_lsst/eups/python/eups/__init__.py", line 5, in ? from cmd import commandCallbacks File "/tmp/test_lsst/eups/python/eups/cmd.py", line 38, in ? import distrib File "/tmp/test_lsst/eups/python/eups/distrib/__init__.py", line 30, in ? from Repositories import Repositories File "/tmp/test_lsst/eups/python/eups/distrib/Repositories.py", line 8, in ? import server File "/tmp/test_lsst/eups/python/eups/distrib/server.py", line 1498 mapping = self._noReinstall if outVersion and outVersion.lower() == "noreinstall" else self._mapping ^ SyntaxError: invalid syntax Perhaps from running the wrong version of python. Full script/log is attached. | 1 |
| 335 | DM-875 | lsst_dm_stack_demo | lsst-dm_stack_demo has obsolete benchmark files (circa Release 7.0) which fail to serve the purpose of validating, for the user, the correct functioning of a freshly built Release v8.0 stack. At the very least, the benchmark files should be regenerated for each official Release. Tasks: (1) Build the benchmark files for Release v8.0 (2) Debate (a) recommending the use of 'numdiff' to check if the output is within realistic bounds. Or, (b) develop another procedure to better show how the current algorithms compare to the algorithms used at the benchmarked Release. (3) Depending on result of the debate on #2: for: (a) provide appropriate 'numdiff' command invocation in manual.; for (b) implement the new procedure. | 40 |
| 336 | DM-882 | Segmentation fault from writing dotted FITS header keywords | Observed on HSC, but I'm not aware of any fix for this on the LSST side either. {code:sh} pprice@tiger3:~ $ gdb python GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1) (gdb) r Starting program: /tigress/HSC/products-20130212/Linux64/python/2.7.6/bin/python >>> from lsst.afw.image import ExposureF >>> exp = ExposureF(1,1) >>> exp.getMetadata().add("A.B.C.D", 12345) >>> exp.writeFits("test.fits") Program received signal SIGSEGV, Segmentation fault. lsst::daf::base::PropertySet::combine (this=0x1eda370, source=...) at src/PropertySet.cc:746 746 if (dj == _map.end()) { (gdb) w #0 lsst::daf::base::PropertySet::combine (this=0x1eda370, source=...) at src/PropertySet.cc:746 #1 0x00002aaabac068dc in lsst::daf::base::PropertyList::deepCopy (this=0x1ddd4c0) at src/PropertyList.cc:74 #2 0x00002aaab7ba559e in lsst::afw::image::MaskedImage<float, unsigned short, float>::writeFits (this=0x1f1d8f0, fitsfile=..., metadata=<value optimized out>, imageMetadata=..., maskMetadata=..., varianceMetadata=...) at src/image/MaskedImage.cc:569 #3 0x00002aaab7b0a7f0 in lsst::afw::image::Exposure<float, unsigned short, float>::writeFits (this=0x1f1d8d0, fitsfile=...) at src/image/Exposure.cc:251 #4 0x00002aaab7b0a9fc in lsst::afw::image::Exposure<float, unsigned short, float>::writeFits (this=0x1f1d8d0, fileName="test.fits") at src/image/Exposure.cc:239 #5 0x00002aaab6b91614 in _wrap_ExposureF_writeFits__SWIG_0 (self=<value optimized out>, args=<value optimized out>) at python/lsst/afw/image/imageLib_wrap.cc:160786 #6 _wrap_ExposureF_writeFits (self=<value optimized out>, args=<value optimized out>) at python/lsst/afw/image/imageLib_wrap.cc:29886 … {code} This appears to only affect dotted keywords (which should be supported through use of {{HIERARCH}}, and even if they're not supported, this should never fail with a segfault). | 4 |
| 337 | DM-887 | Optimize template engine used in configuration tool | string.Template and string interpolation are quite weak, some additional feature would be welcomed : - interpolation interprets wrongly "%d" in template files as template - string.template doesn't manage correctly non referenced template parameters present in template files. | 4 |
| 338 | DM-895 | Use an existing qserv_run_dir with a new Qserv instance/binary | Here's what should be added to qserv-configure.py : - edit $QSERV_RUN_DIR/admin/qserv.conf and change Qserv instance dir to current one (which qserv-configure.sh), - check compliance of QSERV_RUN_DIR with new Qserv instance (version check ?) and/or update configuration files, - re-initialize services, if needed, without breaking already loaded data. | 4 |
| 339 | DM-898 | unset BASH_ENV in newinstall.sh or surrounding instructions | Nutbar users such as myself may have BASH_ENV set, which can mess with your carefully constructed shell environments. Unsetting BASH_ENV should help. This could perhaps go in newinstall.sh, or the documentation alongside where we suggest they unset other variables: https://confluence.lsstcorp.org/display/LSWUG/Building+the+v8.0+LSST+Stack+from+Source | 1 |
| 340 | DM-900 | Update PhoSim tutorial to use CatSim for creating instance catalogs | Update the Process PhoSim Images tutorial (https://confluence.lsstcorp.org/display/LSWUG/Process+PhoSim+Images) to use CatSim to generate instance catalogs, once CatSim documentation is available. This will obviate the need for the helper script refCalCat.py. | 2 |
| 341 | DM-903 | SourceDetectionTask should only add flags.negative if config.thresholdParity == "both" | The SourceDetectionTask always adds "flags.negative" to the schema (if provided) but it is only used if config.thresholdParity == "both". As adding a field to a schema requires that the table passed to the run method have that field this is a significant nuisance when reusing the task. Please change the code to only modify the schema if it's going to set it. | 1 |
| 342 | DM-911 | Provide Task documentation for DipoleMeasurementTask | See Summary. | 2 |
| 343 | DM-912 | Provide Task documentation for PsfMatchTask | See Description (it's currently called PsfMatch) | 4 |
| 344 | DM-913 | Provide Task documentation for ImagePsfMatchTask | See summary | 2 |
| 345 | DM-914 | Provide Task documentation for SnapPsfMatchTask | See summary | 2 |
| 346 | DM-915 | Provide Task documentation for AssembleCcdTask | See summary | 4 |
| 347 | DM-916 | Provide Task documentation for IsrTask | See Summary | 4 |
| 348 | DM-926 | Provide Task documentation for SourceDeblendTask | See Summary | 2 |
| 349 | DM-927 | Provide Task documentation for CmdLineTask | See Summary | 4 |
| 350 | DM-928 | Provide Task documentation for RepairTask | See Summary | 4 |
| 351 | DM-929 | How to write your own command line task, including how-to-retarget sub-tasks | Please provide documentation on how to write a command line task, and how to retarget tasks (e.g. reusing bits of IsrTask for a new camera) The documentation should include a complete annotated example. | 4 |
| 352 | DM-930 | Improve install/configuration/tests documentation and migrate it to reST format | This ticket propose to migrate README and README-devel to reST format (see http://sphinx-doc.org/rest.html). The output is located here : http://lsst-web.ncsa.illinois.edu/~fjammes/qserv-doc/ Furthermore this ticket wil integrate Andy S. DM-622 value-added remarks about Qserv embedded documentation. {quote} README.txt needs a bit of formatting, whole "NOTE FOR DEVELOPERS" is one long line which may need scrolling depending on what do you use to read the file, same applies to README-devel.txt The install procedure in README.txt implies that the whole stack has to be installed including eups. If people have some part of it installed already the it would probably be better to reuse existing stack. Shall we spit install instructions into "Install eups (if not installed already)" and "Install qserv"? README-devel.txt says "Once Qserv is installed...", I don't think that we need or want to install whole qserv before we start development (what if qserv is not available yet for the platform I'm trying to test). What probably needed is installed dependencies, and this should be covered by the comments before 'setup -r .' {quote} | 1 |
| 353 | DM-933 | Photometric calibration uses a column "flux" not the specified filter unless a colour term is active | The photometric calibration code uses a field "flux" in the reference catalog to impose a magnitude limit. If a colour term is specified, it uses the primary and secondary filters to calculate the reference magnitude, but if there is no colour term it uses the column labelled "flux" and ignores the filtername. Please change the code so that "flux" is ignored, and the flux associated with filterName is used. | 1 |
| 354 | DM-936 | Replace getXXXKey for slots with returning functorKeys similar to existing compound Keys | Make any changes in the Functor Key capabilities necessary to support getXXXKey, getXXXErrKey, and getXXXFlagKey for the 3 different types of slots. Change these routines in Source.h.m4 to return FunctorKeys for both version 0 and version 1 tables. Then fixup any compilation breaks on the C++ size which this causes. Remove the version 1 specific accessors. I am assigning this to Jim to confirm that the first part is done, then he can assign the remainder to Perry either by reassigning, or as a subtask. | 4 |
| 355 | DM-944 | Package xrootd-4.0.0rc3-qsClient2 with eups | NULL | 1 |
| 356 | DM-945 | Prevent conflict related to non-unique temporary files created during SciSQL install | SciSQL install sometime fails with next message : {code:bash} [1/2] MySqlScript: scripts/install.mysql [2/2] MySqlScript: scripts/demo.mysql Running testHtm : OK Running testSelect : OK Running testAngSep.py : OK Running testMedian.py : OK Running testPercentile.py : OK Running testS2CPoly.py : OK Running testS2PtInBox.py : OK Running testS2PtInCircle.py : OK Running testS2PtInEllipse.py : OK Running docs.py : FAIL [see /usr/local/home/salnikov/qserv-run/u.fjammes.DM-622-g86a30ec72a/tmp/scisql-0.3.2/build/tools/docs.log] 9 tests passed, 1 failed, and 0 failed to run One or more sciSQL unit tests failed make: *** [install] Error 1 -- CRITICAL: Error code returned by command : / u s r / l o c a l / h o m e / s a l n i k o v / q s e r v - r u n / u . f j a m m e s . D M - 6 2 2 - g 8 6 a 3 0 e c 7 2 a / t m p / c o n f i g u r e / s c i s q l . s h $ cat /usr/local/home/salnikov/qserv-run/u.fjammes.DM-622-g86a30ec72a/tmp/scisql-0.3.2/build/tools/docs.log rm: cannot remove `/tmp/scisql_demo_ccds.tsv': Operation not permitted Failed to run documentation example: rm -f /tmp/scisql_demo_ccds.tsv ERROR 1086 (HY000) at line 4: File '/tmp/scisql_demo_ccds.tsv' already exists {code} It looks like test uses non-unique temporary file name and someone already tried to run installation on the same host. | 2 |
| 357 | DM-946 | Improve management of qserv-run-dir in start/stop scripts. | qserv-start.sh by default tries again to use /usr/local/home/salnikov/qserv-run/2014_05.0 directory. There may be a confusion if we have multiple run directories. Would it be better to install qserv start/stop scripts into run directory itself so that they don't need to guess anything? | 2 |
| 358 | DM-951 | Add Doxygen documentation on rebuilds | Master-branch doxygen documentation should be rebuild on every full master build. | 20 |
| 359 | DM-953 | Move qserv-testdata.py to qserv_testdata package | Andy S. suggests this during DM-622 review, but this needs to be studied deeply, as qserv_testdata goal is to store large datafile, difficult to version in git. | 3 |
| 360 | DM-954 | Clearer and shorter output for qserv-configure.py | qserv-configure.py produces a lot of output which could be confusing if people try to look at it and understand it. It may be better to reduce it to something that just indicates that each step is completed successfully. | 1 |
| 361 | DM-956 | Buildbot email should state if the build used master only or included other branches | Buildbot build status report currently doesn't state if the build was only for master or included other branches requested by the user. | 4 |
| 362 | DM-957 | Use aliases to clean up table version transition | The addition of schema aliases on DM-417 should allow us to clean up some of the transitional code added on DM-545, as we can now alias new versions of fields to the old ones and vice versa. | 2 |
| 363 | DM-958 | Move table versions to Schema | We've added a version number to afw::table::BaseTable to help with the transition to a new approach with different naming conventions and FunctorKeys instead of compound keys. However, it looks like Schema objects need to know about this version number as well, in order to change Subschema objects to split/join using underscores instead of periods. That means we'll have to move the version down into the schema object, which may affect a lot of downstream code. Other than touching a lot of code, the changes should be trivial. | 4 |
| 364 | DM-963 | measAlg.interpolateOverDefects doesn't accept a python list of Defects | The python binding of interpolateOverDefects should accept a python list of Defects as well as a swig-wrapped std::vector<Defect>; it doesn't (try using a python list in test818 in meas_algorithms/tests/Interp.py) | 1 |
| 365 | DM-964 | Include aliases in Schema introspection | Schema stringification and iteration should include aliases somehow. Likewise the extract() Python methods. | 1 |
| 366 | DM-966 | fix int/long conversion on 32-bit systems and selected 64-bit systems | tests/wrap.py fails in pex_config on 32-bit systems and some 64-bit systems (including Ubuntu 14.04) with the following: {code:no-linenum} tests/wrap.py ...EE.E. ====================================================================== ERROR: testDefaults (__main__.NestedWrapTest) Test that C++ Control object defaults are correctly used as defaults for Config objects. ---------------------------------------------------------------------- Traceback (most recent call last): File "tests/wrap.py", line 89, in testDefaults self.assert_(testLib.checkNestedControl(control, config.a.p, config.a.q, config.b)) File "/home/boutigny/CFHT/stack_5/build/pex_config/tests/testLib.py", line 987, in checkNestedControl return _testLib.checkNestedControl(*args) TypeError: in method 'checkNestedControl', argument 2 of type 'double' ====================================================================== ERROR: testInt64 (__main__.NestedWrapTest) Test that we can wrap C++ Control objects with int64 members. ---------------------------------------------------------------------- Traceback (most recent call last): File "tests/wrap.py", line 95, in testInt64 self.assert_(testLib.checkNestedControl(control, config.a.p, config.a.q, config.b)) File "/home/boutigny/CFHT/stack_5/build/pex_config/tests/testLib.py", line 987, in checkNestedControl return _testLib.checkNestedControl(*args) TypeError: in method 'checkNestedControl', argument 2 of type 'double' ====================================================================== ERROR: testReadControl (__main__.NestedWrapTest) Test reading the values from a C++ Control object into a Config object. ---------------------------------------------------------------------- Traceback (most recent call last): File "tests/wrap.py", line 82, in testReadControl config.readControl(control) File "/home/boutigny/CFHT/stack_5/build/pex_config/python/lsst/pex/config/wrap.py", line 212, in readControl __at=__at, __label=__label, __reset=__reset) File "/home/boutigny/CFHT/stack_5/build/pex_config/python/lsst/pex/config/wrap.py", line 217, in readControl self.update(__at=__at, __label=__label, **values) File "/home/boutigny/CFHT/stack_5/build/pex_config/python/lsst/pex/config/config.py", line 515, in update field.__set__(self, value, at=at, label=label) File "/home/boutigny/CFHT/stack_5/build/pex_config/python/lsst/pex/config/config.py", line 310, in __set__ raise FieldValidationError(self, instance, e.message) FieldValidationError: Field 'a.q' failed validation: Value 4 is of incorrect type long. Expected type int For more information read the Field definition at: File "/home/boutigny/CFHT/stack_5/build/pex_config/python/lsst/pex/config/wrap.py", line 184, in makeConfigClass fields[k] = FieldCls(doc=doc, dtype=dtype, optional=True) And the Config definition at: File "/home/boutigny/CFHT/stack_5/build/pex_config/python/lsst/pex/config/wrap.py", line 131, in makeConfigClass cls = type(name, (base,), {"__doc__":doc}) ---------------------------------------------------------------------- Ran 8 tests in 0.017s FAILED (errors=3) {code} There is a partial fix on u/jbosch/intwrappers; this seems to work for Ubuntu 14.04, but not on 32-bit systems. | 2 |
| 367 | DM-967 | qserv-configure.py is broken in master | It looks like there was a bug introduced either during the merge of DM-622 with master or right before that. Running {{qserv-configure.py}} from master fails now: {code} $ qserv-configure.py File "/usr/local/home/salnikov/qserv-master/build/dist/bin/qserv-configure.py", line 229 ("Do you want to update user configuration file (currently pointing ^ SyntaxError: EOL while scanning string literal {code} I assign this to myself, Fabrice is on vacation now and we need to fix this quickly. | 1 |
| 368 | DM-973 | Create a "stub" package that checks for system dependencies for Qserv | Create a special "stab" package that checks and reports missing dependencies. | 2 |
| 369 | DM-975 | setup standard aliases for frequently-used measurements | Frequently-used measurement fields such as classification and pixel flags should have shortened aliases that can be used instead of their full, package-qualified versions. In some cases, these may serve as a sort of slot (e.g. we may have multiple classifications algorithms someday). In this issue, we should audit all current measurement algorithm fields that don't already have a slot that works for them and consider whether there should be a standard alias. We also need to work out a system for defining these aliases, probably in the config for SingleFrameMeasurementTask. | 4 |
| 370 | DM-976 | Detailed documentation for meas_base tasks | We should follow RHL's example for detailed task documentation and document all meas_base tasks. | 2 |
| 371 | DM-977 | Documentation audit and cleanup for meas_base plugins | Many meas_base Plugins and Algorithms have poor documentation, including several whose documentation is a copy/paste relic from some other algorithm. These need to be fixed. | 2 |
| 372 | DM-978 | add base class for measurement tasks | We should consider adding a base class for measurement tasks (SingleFrameMeasurementTask, ForcedMeasuremedTask) that includes the callMeasure methods. I'm hoping this will help cleanup callMeasure and improve code reuse. | 1 |
| 373 | DM-979 | Add FunctorKey to replace Coord compound keys | Unlike previous FunctorKey replacements, Coord compound Keys are used in the minimal schema for SimpleTable and SourceTable, which makes removing them problematic. | 1 |
| 374 | DM-980 | convert measurement algorithms in ip_diffim | ip_diffim includes a few measurement algorithms which need to be converted to the new framework. | 5 |
| 375 | DM-981 | convert measurement algorithms in meas_extensions_shapeHSM | This is a low-priority ticket to replace the old-style plugins in meas_extensions_shapeHSM with new ones compatible with meas_base. As this isn't a part of the main-line stack, we should delay it until other the meas_base conversion is nearly (or perhaps fully) complete. | 3 |
| 376 | DM-982 | convert meas_extensions_photometryKron to new measurement framework | This is a low-priority ticket to replace the old-style plugins in meas_extensions_photometryKron with new ones compatible with meas_base. As this isn't a part of the main-line stack, we should delay it until other the meas_base conversion is nearly (or perhaps fully) complete. | 3 |
| 377 | DM-983 | Finish the Log packaging | Finish the work started by Bill (DM-778) | 6 |
| 378 | DM-984 | allow partial measurement results to be set when error flag is set | We need to be able to return values at the same time that an error flag is set. The easiest way to do this is to have Algorithms take a Result object as an output argument rather than return it. We'll revisit this design later. | 2 |
| 379 | DM-989 | .my.cnf in user HOME directory breaks setup script | Presence of {{.my.cnf}} file in the user HOME directory crashes {{qserv-configure.py}} script if parameters in {{.my.cnf}} conflict with parameters in {{qserv.conf}}. How to reproduce: * create .my.cnf file in the home directory: {code} [client] user = anything # host/port and/or socket host = 127.0.0.1 port = 3306 socket = /tmp/mysql.sock {code} * try to run {{qserv-configure}}, it fails with error: {code} /usr/local/home/salnikov/qserv-run/u.salnikov.DM-595/tmp/configure/mysql.sh: connect: Connection refused /usr/local/home/salnikov/qserv-run/u.salnikov.DM-595/tmp/configure/mysql.sh: line 13: /dev/tcp/127.0.0.1/23306: Connection refused ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) {code} It looks like {{~/.my.cnf}} may be a left-over from some earlier qserv installation. If I remove it and re-run {{qserv-configure.py}} now it's not created anymore. Maybe worth adding some kind of protection to {{qserv-configure.py}} in case other users have this file in their home directory. | 2 |
| 380 | DM-991 | add query involving a blob to the integration tests | We need to add a query (or more?) to the qserv_testdata that involve blobs. Blobs are interesting because they might break some parts of the qserv if we failed to escape things properly etc. | 2 |
| 381 | DM-993 | improve message from qserv_testdata | Currently, when I try to run qserv-benchmark but qserv_testdata was not setup, I am getting {code} CRITICAL Unable to find tests datasets. -- FOR EUPS USERS : Please run : eups distrib install qserv_testdata setup qserv_testdata FOR NON-EUPS USERS : Please fill 'testdata_dir' value in ~/.lsst/qserv.conf with the path of the directory containing tests datasets or use --testdata-dir option. {code} It is important to note in the section for eups users that this has to be called BEFORE qserv is setup, otherwise it has no effect. | 1 |
| 382 | DM-994 | make slot config validation more intelligent | Slot config validation currently assumes that field names match plugin names, which is not always a safe assumption. This can prevent certain algorithms from being used in slots. We probably can't do this validation in Config.validate(); we need to check in the measurement Task constructor after the schema has already been constructed. | 1 |
| 383 | DM-995 | Make NoiseReplacer a context manager | I think the API for NoiseReplacer could be made more idiomatic (and possibly safer) by turning it into a "context manager" (i.e. so it can be used with the "with" statement). | 1 |
| 384 | DM-998 | Adapt integration test to multi-node setup v1 | Following DM-595 we can start qserv in multi-node configuration. Next step is to be able to run integration tests in that setup. This needs a bit of understanding how to distribute chunks between all workers in a cluster and how to load data in remote mysql server. | 10 |
| 385 | DM-999 | rename config file(s) in Qserv | Rename local.qserv.cnf to qserv-czar.cnf. It is quite likely there are some other config files that would make sense to rename. If you see some candidates, let's discussion on qserv-l and do the renames. | 1 |
| 386 | DM-1001 | Modify assertAlmostEqual in ip_diffim subtractExposures.py unit test | In unit test, the comparison self.assertAlmostEqual(skp1[nk][np], skp2[nk][np], 4) fails. However if changed to self.assertTrue(abs(skp1[nk][np]-skp2[nk][np]) < 10**-4) which is the desired test, this succeeds. This ticket will remove all assertAlmostEquals from subtractExposure.py and replace with the fundamental comparison operator of the absolute value of the differences. | 1 |
| 387 | DM-1002 | Research Apache Mesos and Google Kubernetes | It'd be good to check out Apache Mesos and Google Kubernetes and see how relevant they are for Qserv / LSST Database | 4 |
| 388 | DM-1004 | Provide Task documentation for ModelPsfMatchTask | See Description (it's currently called PsfMatch) | 2 |
| 389 | DM-1005 | Migrate Qserv worker code to the new logging system | NULL | 8 |
| 390 | DM-1010 | fix names of meas_base plugins to match new naming standards | Some meas_base plugins still have old-style algorithm names. | 1 |
| 391 | DM-1011 | Remove use of compound fields in minimal schema | If we want to ultimately remove compound fields, we need to remove the "coord" field from the SimpleTable/SourceTable minimal schema, and provide a different way to get ra,dec from a source. | 4 |
| 392 | DM-1012 | remove temporary workaround in new SkyCoord algorithm | SingleFrameSkyCoordPlugin is using the Footprint Peak, not the centroid slot. According to comments in the code, this is a workaround for some problem with centroids. This needs to be fixed. | 1 |
| 393 | DM-1013 | Classification should set flags upon failure | The classification algorithm claims it can never fail. It can, and should report this. | 2 |
| 394 | DM-1014 | Detailed documentation for GaussianCentroid | We need more detailed documentation for the GaussianCentroid algorithm, in terms of how it actually computes the centroid. We (Jim and Perry) have done what we can, but we need help from whoever actually wrote it (RHL, we think) to provide the rest. In particular: - Additional detail should be filled in in the class Doxygen for GaussianCentroidAlgorithm, in GaussianCentroid.h - The "noPeak" flag field description and name should be compared to what the algorithm actually does with it. It looks to me like it's a bit misnamed (and maybe shouldn't be considered an error condition at all, if we want to run this on difference images), but I'm not sure. | 1 |
| 395 | DM-1015 | convert GaussianFlux to use shape, centroid slots | We should cleanup and simplify the GaussianFlux algorithm to simply use the shape and centroid slot values instead of either computing its own or having configurable field names for where to look these up. | 1 |
| 396 | DM-1016 | Detailed documentation for SdssCentroid | We needed detailed documentation for how SdssCentroid actually works. I think it involves fitting something like a symmetrized PSF model, but I'm sufficiently unsure of the details that RHL should write this, not me. It should go in the Doxygen class docs for SdssCentroidAlgorithm in SdssCentroid.h in meas_base. | 1 |
| 397 | DM-1017 | fix testForced.py | testForced.py is currently passing even though it probably should be failing: it's trying to get centroid values from a source which has neither a valid centroid slot or a Footprint with Peaks (I suspect because transforming a footprint might remove the peaks). Prior to DM-976, that would have caused a segfault; on DM-976, I've turned it into an exception, which is then turned into a warning by the measurement framework. | 2 |
| 398 | DM-1018 | Fix incorrect eupspkg config for astrometry_net | The clang patch from 8.0.0. version was (correctly) deleted. However, the patch identity was still left in the eupspkg config's protocol. This will delete the last vestige of the formerly necessary clang patch. | 2 |
| 399 | DM-1022 | fix warnings related to libraries pulled through dependent package | This came up during migrating qserv to the new logging system, and it can be reproduced by taking log4cxx, see DM-983, essentially: {code} eups distrib install -c log4cxx 0.10.0.lsst1 -r http://lsst-web.ncsa.illinois.edu/~becla/distrib -r http://sw.lsstcorp.org/eupspkg {code} cloning log package (contrib/log.git), building it and installing in your stack, and finally taking the branch u/jbecla/DM-207 of qserv and building it. The warnings looks like: {code}/usr/bin/ld: warning: libutils.so, needed by /usr/local/home/becla/qservDev/Linux64/log/1.0.0/lib/liblog.so, not found (try using -rpath or -rpath-link) /usr/bin/ld: warning: libpex_exceptions.so, needed by /usr/local/home/becla/qservDev/Linux64/log/1.0.0/lib/liblog.so, not found (try using -rpath or -rpath-link) /usr/bin/ld: warning: libbase.so, needed by /usr/local/home/becla/qservDev/Linux64/log/1.0.0/lib/liblog.so, not found (try using -rpath or -rpath-link) {code} and they show up when I build qserv package, and are triggered by the liblog. I suspect sconsUtils deal with that sort of issues, but since we have our own scons system for qserv it is not handled. Fabrice, can you try to find a reasonable solution for that? Thanks! | 1 |
| 400 | DM-1026 | make use of MeasurementDataFlags | We started adding a system to allow algorithms to declare what kind of data they can run on, but never really put it in place. To do this, we should: - Add more flags (at least NO_CALIB). - Pass these flags in pipe_tasks and other places with tasks that use measurement tasks as subtasks (for instance, we should set NO_WCS and NO_CALIB during calibrate.initialMeasurement, and COADD in processCoadd). - Add checks for these flags in appropriate algorithms. For instance, PsfFlux should fail in construction if NO_PSF is set, and PeakLikelihoodFlux should fail if PRECONVOLVED is not set. | 6 |
| 401 | DM-1028 | qserv-version.sh produces incorrect version number | I have just installed qserv on a clean machine (this is in a new virtual machine running Ubuntu12.04) which got me version 2014_07.0 installed: {code} $ eups list qserv 2014_07.0 current b76 $ setup qserv $ eups list qserv 2014_07.0 current b76 setup $ echo $QSERV_DIR /opt/salnikov/STACK/Linux64/qserv/2014_07.0 {code} but the {{qserv-version.sh}} script still thinks that I'm running older version: {code} $ qserv-version.sh 2014_05.0 {code} | 2 |
| 402 | DM-1029 | "source" command is not in standard shell | {{qserv-start.sh}} script fails when installed on Ubuntu12.04: {code} $ ~/qserv-run/2014_05.0/bin/qserv-start.sh /home/salnikov/qserv-run/2014_05.0/bin/qserv-start.sh: 4: /home/salnikov/qserv-run/2014_05.0/bin/qserv-start.sh: source: not found /home/salnikov/qserv-run/2014_05.0/bin/qserv-start.sh: 6: /home/salnikov/qserv-run/2014_05.0/bin/qserv-start.sh: check_qserv_run_dir: not found {code} It complains about {{source}} command. {{source}} is not standard POSIX shell command, it is an extension which exists in many shells. Apparently in older Ubuntu version {{/bin/sh}} is stricter about non-standard features. To fix the script one either has to use standard . (dot) command or change shebang to {{#!/bin/bash}}. This of course applies to all our executable scripts. | 2 |
| 403 | DM-1030 | W15 Refactoring of Qserv | Refactoring Qserv code to make it cleaner, better, and most importantly more robust and resilient to failures. JK: Refer to loading spreadsheet for PMCS assignments | 84 |
| 404 | DM-1031 | W15 Integration Testing of Qserv | Improvements to the integration tests suite for Qserv, including making it more generic (it currently uses a hardcoded database name), integrating new partitioner, and adding a data set from one of the last data releases. JK: Refer to loading spreadsheet for PMCS assignments | 19 |
| 405 | DM-1033 | W15 Design Metadata Store for production tracking (v1) | Come up with a design of Metadata Store that will track all information in the Science Data Archive (across Data Releases), for both image and database repositories JK: Refer to loading spreadsheet for PMCS assignments | 45 |
| 406 | DM-1034 | W15 Initial version of image and file archive | Build initial (alpha) version of system for tracking existing image data sets. (a) Build a web form where users enter information about existing data sets. (b) Design and build a MySQL backend (maybe through DataCat). (c) Implement crawler that automatically fetches metadata from FITS headers for data set registered by users. Note that support for additional formats (eg config files) will be added in the future, not through this epic. Note that this initial version will not be production ready, we will make it more bullet-proof and feature rich during S15 cycle. JK: Refer to loading spreadsheet for PMCS assignments | 30 |
| 407 | DM-1035 | W15 Butler (v2) for local data sets | Improvements and tweaks to the butler as needed based on the feedback from the Apps team JK: Refer to loading spreadsheet for PMCS assignments | 19 |
| 408 | DM-1036 | W15 Management of distributed Qserv databases and tables | This includes tasks such as creating and deleting distributed databases managed by Qserv, creating and deleting partitioned tables based on the information that is stored in the CSS. Design and exploratory prototype. JK: Refer to loading spreadsheet for PMCS assignments | 39 |
| 409 | DM-1037 | W15 Loading data into distributed Qserv-managed databases | Improvements to the existing data loader, to simplify data loading into Qserv. That includes integration with information in the CSS. This version will include non-parallel, single-node loader. Relevant document (describing a sketch of much more advanced version): https://dev.lsstcorp.org/trac/wiki/db/Qserv/DataLoading JK: Refer to loading spreadsheet for PMCS assignments | 28 |
| 410 | DM-1038 | S15 Implement Query Mgmt in Qserv | Initial version of system for managing queries run through qserv. This includes capturing information about queries running in Qserv. Note, we are not dealing with query cost estimate here, (it will be covered through DM-1490). | 40 |
| 411 | DM-1041 | eliminate confusing config side-effects in CalibrateTask | CalibrateTask does some unexpected things differently if you configure it certain ways, because it perceives certain processing as only being necessary to feed other steps. In particular, if you disable astrometry and photometric calibration, it only runs measurement once, because it assumes the only purpose of the post-PSF measurement is to feed those algorithms. This (as well as poor test coverage) made it easy to break CalibrateTask in the case where those options are disabled a few branches back. After conferring with Simon and Andy, we think the best solution is to remove this sort of conditional processing from CalibrateTask, which should also make it much easier to read. Instead, we'll always do both the initial and final phase of measurement, even if one of those phases is not explicitly being used within CalibrateTask itself. | 1 |
| 412 | DM-1045 | Create a permanent and accessible mapping of the BB# and the bNNN. | Create a permanent and accessible mapping of the BB# and the bNNN. The users are interested in the BB# since is is used to point to the STDIO file form the entire stack build. The bNNN is needed because the daily life of the developer revolves around the stack tagged alternately by the bNNN tags and/or the DM Release tags. | 2 |
| 413 | DM-1046 | W15 Central State System, support for db/table/query metadata | Improvements to the CSS system to make it more thread safe, more compact representation of keys, and adding information that will be needed to handle distributed databases, tables and likely queries (if we decided to use CSS for Query management). JK: Refer to loading spreadsheet for PMCS assignments | 49 |
| 414 | DM-1051 | W15 Image Cutout Service | First version of the images cutout service. It will rely on butler. The bulk of the work in this epic includes building a web-based front-end. JK: Refer to loading spreadsheet for PMCS assignments | 19 |
| 415 | DM-1052 | Add / improve tests and examples for Log package | NULL | 1 |
| 416 | DM-1054 | init.d/qserv-czar needs LD_LIBRARY path | With the addition of log we now need to find some shared libraries from stack. Current version of qserv-czar init.d script does not capture LD_LIBRARY_PATH, so we should add it there. | 1 |
| 417 | DM-1055 | Remove unnecessary pieces from qserv czar config | The config file for the qserv czar has some items that are no longer relevant, and in this issue, we focus on the ones that are clearly the responsibility of our qserv css. This ticket includes: -- removing these items from the installation/configuration templates -- removing these items from sample configuration files -- removing these items from the code that reads in the configuration file and sets defaults for these items -- fixing things that seem to break as a result of this cleanup. danielw volunteers to assist on the last item, as needed. | 2 |
| 418 | DM-1058 | fix SubSchema handling of "." and "_" | SubSchema didn't get included in the rest of the switch from "." to "_" as a field name separator. As part of fixing this, we should also be able to simplify the code in the slot definers in SourceTable. | 1 |
| 419 | DM-1059 | track down difference in SdssShape implementation | The meas_base version of SdssShape produces slightly different outputs from the original version in meas_algorithms, but these should be identical. We should understand this difference rather than assume its benign just because it's small. | 2 |
| 420 | DM-1060 | S15 Data Distribution & Replica Mgmt Design | This is a continuation of DM-779, continue research related to data distribution and replication. There are still many option questions: should it be centralized or peer-to-peer? Is bittorent useful? How much should be home grown vs off-the-shelf (and which off-the-shelf). There are no direct deliverables in FY2015, but it is a complex topic, so we should not wait until the last minute. This epic covers the R&D / design. | 100 |
| 421 | DM-1061 | W15 Qserv Stability / bug fixes | Tweaks, improvements, fixes to bugs discovered over the course of W15, to improve Qserv stability. JK: Refer to loading spreadsheet for PMCS assignments | 83 |
| 422 | DM-1067 | move algorithm implementations out of separate subdirectory | We should move the code in the algorithms subdirectory (and namespace) into the .cc files that correspond to individual algorithms. They should generally go into anonymous namespaces there. After doing so, we should do one more test to compare the meas_base and meas_algorithms implementations. | 1 |
| 423 | DM-1068 | audit and clean up algorithm flag and config usage | Check that meas_base plugins and algorithms have appropriate config options and flags (mainly, check that there are no unused config options or flags due to copy/paste relics). | 1 |
| 424 | DM-1070 | switch default table version to 1 | Now that all tasks that use catalogs explicitly set the table version, it should be relatively straightforward to set the default version to 1 in afw. Code that cannot handle version > 0 tables should continue to explicitly set version=0. | 2 |
| 425 | DM-1071 | Switch default measurement tasks to meas_base | We should set the default measurement task in ProcessImageTask to SingleFrameMeasurementTask, and note that SourceMeasurementTask and the old forced photometry drivers are deprecated. | 2 |
| 426 | DM-1072 | create forced wrappers for algorithms | We have multiple algorithms in meas_base which could be used in forced mode but have no forced plugin. We should go through the algorithms we have implemented and create forced plugin wrappers for these. | 1 |
| 427 | DM-1073 | remove old forced photometry tasks | After meas_base has been fully integrated, remove the old forced photometry tasks from pipe_tasks | 1 |
| 428 | DM-1074 | Measurement - Calibration and Ingest | Create command-line tasks to transform raw measurement quantities (e.g. fluxes, positions in pixels) to global units (e.g magnitudes and positions in celestial coordinates). Also requires interfacing with the measurement plugin system, as only plugins can be authoritative on how to transform their outputs. JK: In PMCS this would be Bosch J 50% Gee P 50% | 20 |
| 429 | DM-1076 | convert afw::table unit tests to version 1 | Most afw::table unit tests explicitly set version 0. We should change these to test the new behaviors, not the deprecated ones. | 2 |
| 430 | DM-1077 | Audit TCT recommendations to ensure that all standards updates were installed into Standards documents. | Audit TCT recommendations to ensure that all standards updates were installed into Standards documents. It was found that the meeting recorded in: [https://dev.lsstcorp.org/trac/wiki/Winter2012/CodingStandardsChanges] failed to include two recommendations: * recommended: 3-30: I find the Error suffix to be usually more appropriate than Exception. ** current: 3-30. Exception classes SHOULD be suffixed with Exception. * recommended but not specifically included: Namespaces in source files: we should use namespace blocks in source files, and prefer unqualified (or less-qualified) names within those blocks over global-namespace aliases. ** Rule 3-6 is an amalgam of namespace rules which doesn't quite have the particulars desired. FYI: The actual vote was to: "Allow namespace blocks in source code (cc) files." To simplify the future audit, all other recommendations in that specific meeting were verified as installed into the standards. | 2 |
| 431 | DM-1078 | add batch flag to newinstall.sh | In some cases (installing in some script environment) it is nice to be able to run without any interaction. This will add a flag to tell the script to install without asking about anaconda and git. It will assume the answer is 'yes' to any question it would have asked. | 1 |
| 432 | DM-1079 | Organize the content of the DM Developer Guide | The DM Developer Guide is presently a place-holder for information relevant to developers. It is time to re-organize the content, and to provide authoritative guidance to developers. This will involve contributions from various members of the DM Team, and likely some selected harvesting of the Trac/Wiki. | 10 |
| 433 | DM-1083 | Fix overload problems in SourceCatalog.append and .extend | This example fails with an exception: {code:py} import lsst.afw.table as afwTable schema = afwTable.SourceTable.makeMinimalSchema() st = afwTable.SourceTable.make(schema) cat = afwTable.SourceCatalog(st) tmp = afwTable.SourceCatalog(cat.getTable()) cat.extend(tmp) {code} Expected behavior is that the last line is equivalent to {{cat.extend(tmp, deep=False)}}. | 1 |
| 434 | DM-1085 | Update the DMS and Astro Glossaries | The DMS and Astro Glossaries in Confluence define a set of technical terms used in their respective domains. Some of the definitions are placeholders, and other terms used in the SWUG, DM Space, and DM Developer Guide have yet to be defined in one glossary or the other. It is time to update these docs. | 2 |
| 435 | DM-1086 | Determine problem with Mac OS X processCCD output when compared to 'identical' dataset generated on RHEL6 | The benchmark file for the summer2012 demo is generated on RHEL6. When using the dataset to compare stack output generated on Mac OSX 10.8.5 (and 10.9) there are significant deviations. Find out where the problem arises...in the stack during processing or in the comparison. | 4 |
| 436 | DM-1087 | Research how to kill query in mysql | In the port to the new Xrootd Ssi API, worker-side squashing was lost in the shuffle. The plumbing is different, and re-implementing squash functionality is not entirely straightforward, especially because the new API is still missing documentation and examples for implementing cancellation. The consequences of not implementing this are minor--some extra work may be done by the worker, but not a whole lot, because user-level cancellation has not been implemented. First step is to understand how mysql code is handling query killing. | 4 |
| 437 | DM-1088 | Investigate HTCondor config settings to control speed of ClassAd propagation | With default settings we do not have good visibility as to whether an updated ClassAd on a compute node (e.g., CacheDataList now has ccd "S00") will be in effect on the submit node in time for a Job to be matched to an optimal HTCondor node/slot. There are several components (negotiator, schedd, startd) and their associated activities that could impact the time that it takes for a new ClassAd on a worker node to 'propagate' back to the submit side. We investigate these configuration settings to try to determine what thresholds for configuration settings are required to meet a given time cadence of job submissions. | 2 |
| 438 | DM-1089 | Enhance SWUG chapter on measurement | A basic description of measurement in the LSST Stack was created for the SWUG (see DM-692). However, this chapter lacks specifics about how the measurement algorithms work and does not yet mention some of the available algoritms. This task is intended to provide the next level of detail, with references to source-level descriptions that exist or are in progress. | 6 |
| 439 | DM-1099 | afw::table - finish interface transition | JK: In PMCS Bosch J and Gee P | 36 |
| 440 | DM-1100 | Measurement - Convert Old Algorithms | JK: In PMCS Bosch J and Gee P | 31 |
| 441 | DM-1101 | Measurement - Finish Framework Overhaul | JK: In PMCS Bosch J and Gee P | 51 |
| 442 | DM-1103 | Design unit-test data package | Gather requirements for unit test datasets and determine how to organize and generate the data. | 4 |
| 443 | DM-1104 | Generate test data | Write scripts to generate the test data (as needed) and run them. | 10 |
| 444 | DM-1105 | Custom mapper for unit test data package | We need a custom mapper for the test data package, so we can run unit tests against it without depending on any particular obs_* package. | 6 |
| 445 | DM-1106 | Convert existing tests to use new package | After the new unit test data package is available, we should convert all tests that currently use afwdata or obs_test to use the new package instead, and then remove obs_test and afwdata. Should have subtasks and a re-estimate of effort after auditing how much there is to be done. | 30 |
| 446 | DM-1107 | afw - Footprint Improvements | This epic contains several improvements to the Footprint class, including API refactoring, performance-related reimplementations, and some new features. JK: In PMCS this would be Bosch J and Swinbank J | 30 |
| 447 | DM-1108 | Galaxy Fitting - Shear Precision Experiments | NB some of the below stories will actually be Lauren, not Perry. Breakdown: pgee 58%; jbosch 21%; krughoff 6%; lauren 15% | 100 |
| 448 | DM-1109 | Minimal MultiFit system prototype | Experiment with a framework for running "multifit"-style fitting algorithms (i.e. fit a model convolved with the PSF to multiple exposures simultaneously). Earlier prototypes (meas_multifit) have demonstrated executable multifit code on a single system: repeating that work is not the priority here. Rather, we will produce a concept for the framework for running multifit at scale using large datasets on a cluster. It is not a requirement that it be possible to run an at-scale multifit job and produce useful results by the end of this epic; however, there should be proof-of-concept code which demonstrates the structures within which future multifit work will be carried out. This will be supplied to both Science Pipelines and Process Middleware developers. | 100 |
| 449 | DM-1110 | Galaxy Fitting - Sampling Algorithm Plugin | NULL | 60 |
| 450 | DM-1111 | CModel robustness | The CModel code is currently a mixture of work ported from HSC and some new development on LSST. It needs to be validated on real data (HSC, SDSS, CFHT). All known failure modes should be eliminated. Where possible, improve its performance. | 65 |
| 451 | DM-1112 | Create utilities to allow camera testing team to use CameraGeom | The camera team can use the CameraGeom classes to reduce lab data for testing purposes. Since the camera is relatively flexible, a good way of doing this is to create a camera at runtime from the header keys in the files to be reduced. JK: In PMCS this would be Krughoff S | 30 |
| 452 | DM-1113 | Make the API for ISR explicit | The run method of the IsrTask currently takes a dataRef which has getters for calibration products. This makes the task hard to re-use because one needs a butler and because the interface is opaque. This task will make the IsrTask API more transparent. JK: In PMCS this would be Krughoff S | 20 |
| 453 | DM-1114 | Re-write astrometry task to remove dependency on astrometry.net | The astrometry task currently depends as astrometry.net as the only possible algorithmic back end. This also has created a dependency on the astrometry.net file format. This task will remove the dependency on astrometry.net file format and will implement the current HSC astrometry solver in the astrometry task. This will focus on solving single chips, not the full focal plane. Note that DM-167 (which was closed with reference to this ticket) requests that we split photometric and astrometric catalogues, and this is still needed. JK: In PMCS this would be Krughoff S and Owen R | 80 |
| 454 | DM-1115 | Command-line Tasks - Unit tests for ProcessCcd | We can get a long way toward improved test coverage in pipe_tasks by having Unit tests for ProcessCcd. JK: In PMCS this would be Owen R | 20 |
| 455 | DM-1116 | Add numpy typemaps to afw interfaces so numpy types pass correctly to C++ | Currently SWIGed interfaces do not handle numpy types correctly. This can be fixed with the appropriate typemap. JK: In PMCS this would be Owen R | 11 |
| 456 | DM-1117 | Extend Exposure classes to contain background models | Exposures carry many things: PSFs, FITS metadata, Detectors, etc. Another object algorithms may want to access with an exposure is the set of background models subtracted from the exposure to that point. This will involve porting afw.math.BackrgoundList to C++ and extending Exposure classes to contain the new BackgroundList object. JK: In PMCS this would be Krughoff S | 20 |
| 457 | DM-1118 | Improve documentation of code from DM-70 | DM-70 was merged before its code documentation was completely ready, in an effort to not leave such a large amount of code un-merged. This ticket exists as the second half of DM-70 to clean up its documentation. | 6 |
| 458 | DM-1120 | Improve fault tolerance as demonstrated in nightly computing simulator | Adjust simulator parameters, mechanisms, and implementation to minimize time to recover and unprocessed/unarchived data resulting from a machine or network failure. For W15: Pietrowicz, S - 100% Start Oct 2014, finish Nov 2014 For S15 rollover: Pietrowicz, S - 12% Start March 2015, finish April 2015 | 50 |
| 459 | DM-1121 | Simplify and refactor the Event Services API | Remove old fixed metadata; refactor using more generic metadata interface. For W15: Pietrowicz, S - 100% Start Dec 2014, finish Dec 2014 For S15 rollover: Pietrowicz, S - 11% Start March 2015, finish April 2015 | 18 |
| 460 | DM-1122 | Write a log4cxx handler that emits events for log messages | Write a log4cxx handler that emits events for log messages. For W15: Pietrowicz, S - 100% Start Jan 2015, end mid Jan 2015 For S15 rollover: Pietrowicz, S - 11% Start March 2015, finish April 2015 | 18 |
| 461 | DM-1123 | requirements and specifications for initial version of inter-Task communication | gather requirements and specification for Implementation of a gather/scatter mechanism for a set of Tasks executing in parallel using the Event Services, providing for eventual extension to support MPI or other communication mechanisms. Start mid Jan 2015, finish Feb 2015. JK: In PMCS this would be Pietrowicz S | 50 |
| 462 | DM-1124 | Chebyshev approximation object for aperture corrections | Using the interface defined in DM-740, implement a Chebyshev-based implementation to be used to interpolate aperture corrections across CCD-level interfaces. A prototype is available on the HSC fork, which can be copied directly, modulo any interface changes on DM-740. For more information, see the HSC Jira issue (which also includes the work associated with DM-740): https://hsc-jira.astro.princeton.edu/jira/browse/HSC-796 and the HSC git commits https://github.com/HyperSuprime-Cam/afw/compare/releases/S14A_0...tickets/DM-796 | 3 |
| 463 | DM-1125 | avoid usage of measurement framework in star selectors | At least one of the star selectors uses the old measurement framework system to measure the moments of a cloud of points. With the new versions of all the measurement plugins, it should be much easier (and cleaner) to just call the SdssShape algorithm directly, instead of dealing with the complexity of applying the measurement framework to something that isn't really an image. | 3 |
| 464 | DM-1126 | design new Footprint API | This issue is for *planning* (not implementing) some changes to Footprint's interface, including the following: - make Footprint immutable - create a separate SpanRegion class that holds Spans and provides geometric operators does not hold Peaks or a "region" bbox (Footprint would then hold one of these). - many operations currently implemented as free functions should be moved to methods - we should switch the container from vector<PTR(Span)> to simply vector<Span>, as Span is nonpolymorphic and at last as cheap to copy as a shared_ptr. The output of this issue will be a set of header files that define the new interface, signed off by an SAT design review. Other issues will be responsible for implementing the new interface and fixing code broken by the change. | 8 |
| 465 | DM-1127 | implement new Footprint API | This issue is the implementation for DM-1126, including fixing code broken by the API changes. This issue should be given subtasks for discrete pieces of work as part DM-1126, as it has a lot of story points. Because it's likely all of this needs to be merged at the same time, it should probably be done on a single branch, unless some of the earlier work can be done in a backwards-compatible way.. | 20 |
| 466 | DM-1128 | Span-based grow operations for Footprint | The current grow operation for Footprints is very inefficient for isotropic grows. A better algorithm can be found in the attached paper. | 10 |
| 467 | DM-1129 | Span-based topological set operations for Footprint | Implement span-based overlap tests and spatial union, intersection, and difference operators for Footprints. Should be split into subtasks; first task would be to come up with the interface, and then each operation to be implemented should have one subtask each. If complete after DM-1126 and DM-1127, these operations should be implemented in the SpanRegion class, and Footprint should delegate to that. | 20 |
| 468 | DM-1130 | refactor C++ Algorithm classes | Using one of the prototypes of DM-828, refactor the C++ Algorithm code in meas_base and any other packages. | 10 |
| 469 | DM-1131 | create PSF simulations for shapelet approximation test | To determine the number and order of shapelet expansions needed to approximate the LSST PSF, we need simulations of LSST PSFs, presumably generated using PhoSim. I think we want something like 50-100 extremely high-SNR stars, drawn from realistic distributions of anything that would affect the PSF, including position on the focal plane and center position within a pixel. I only care about getting these out as postage stamps, so I'll leave the question of how best to run PhoSim to get these up to someone else. While this is part of a mostly-Princeton Epic, I'm assigning this issue to Simon as he'd be able to do it much faster than I could. I'll also let him review and update the story points estimate. | 6 |
| 470 | DM-1132 | create galaxy simulations for shapelet approximation and truncation tests | Using the PSF simulations generated in DM-1131 and the GalSim package, generate ~20k galaxies for each of 3 shear values for each input PSF. Will need to be divided into subtasks - figure out what the outputs should look like, write the code to generate the simulations, estimate the time need to run the simulations, run the simulations at scale, etc. | 30 |
| 471 | DM-1133 | implement shear measurement driver for simulations | We'll need driver code to run the galaxy fitting algorithms on the simulations from DM-1132. Efficiently saving all the samples could be challenging. | 10 |
| 472 | DM-1134 | implement SDSS PSF residual trick | SDSS galaxy fitting approximated the convolution of a galaxy model with the PSF as the convolution of the galaxy model with a simplified approximation to the PSF added to the difference between the PSF approximation and the true PSF. We should do the same, as it'll be no worse than ignoring the difference between the PSF approximation and the true PSF, and it may greatly reduce the complexity needed in the approximation. | 8 |
| 473 | DM-1135 | test how large pixel region used in galaxy fitting needs to be | Using simulations built on DM-1132 and driver code from DM-1133, test different pixel region sizes and shapes, and determine at what point shear bias due to finite fit region drops below a TBD threshold. | 20 |
| 474 | DM-1136 | test number/order of shapelet expansions needed to approximate PSF | Run driver from DM-1133 on simulations from DM-1132 with different configuration for shapelet PSF approximation, increasing complexity until change in shear estimate drops below a TBD threshold. | 14 |
| 475 | DM-1137 | Evaluate python/c++ documentation generation and publication tools | This epic related to documentation that is provided as part of normal development activities. The desire is to keep this documentation in and near the codebase as this is best practice for it being maintainable. At the other end, we wish to publish this documentation in a coherent and searchable way for users. A number of tools exist in this area and this item requires a preliminary evaluation to be made. This is part of curating our documentation infrastructure. [FE 75% DOC 100% starting August 20th] | 20 |
| 476 | DM-1138 | Demonstrate & iterate with team on documentation toolchain | Following from DM-1137, this epic relates to demonstrating various options for documentation tools workflows to the team, gathering input as to the preferred solution, adopting a workflow, and defining any specific implementation choices. This is part of curating our documentation infrastructure. | 5 |
| 477 | DM-1139 | Stack documentation infrastructure and migration | [Epic retitled and bumped in points to reflect wider scope and change in resource allocation] This epic containes work on migrating the documentation infrastructure into sphinx (from doxygen, Confluence) subject to RFC, continuous deployment of documentation, read-the-docs or similar presentation, standardisation via teplates of common information, a proposal for CI-ing examples and tutorials, release and install note documentation, and visual design and javascript development for UI/UX An additional request to migrate Word-based design documentation has been accepted. LaTeX support is being investigated. [JS 100%] | 63 |
| 478 | DM-1140 | Investigate automatic MacOS X build/deploy | This item is to set up the same continuous integration process we have on Linux on a MacOSX test server. JK: In PMCS this would be Economou F and New Hire LS3 | 4 |
| 479 | DM-1141 | Evaluate merits of alternative CI and RFC | Evaluate whether we wish to continue buildbot development or use a different (or additional) continuous intergration system. This is part of a continuous occasional process of evaluating whether our current toolchain choices are still meeting our needs. [FE at 75%, JH at 75%] | 30 |
| 480 | DM-1142 | Regularise Nightly and Weekly builds | The intent here is to create two seperate automated deployment environments, one based on a nightly (or ad-hoc) build, one one aslower cadence (eg weekly). This will allow us to do intergration/QA runs on a bleeding or trailing edge as required. JK: In PMCS this would be Economou F and New Hire LS3 | 6 |
| 481 | DM-1143 | Investigate candidates for Verification and Integration Data Sets | The task here is to develop a data set that can be used both for continuous integration (build tests) and automatic QA (integration tests). We want to maximise the richness of the data set in terms of its usefulness, but minimise it in terms of its size. DN to co-ordinate contributions. [DN 95% FE 5%] | 40 |
| 482 | DM-1145 | Remove code made obsolete | NULL | 4 |
| 483 | DM-1147 | Create a top-level qserv_distrib package | qserv_distrib will be a meta-package embedding qserv, qserv_testdata and partition. | 2 |
| 484 | DM-1148 | SUI: Research system framework for SUI development | Current IPAC development utilizes GWT, is it the right system for LSST SUI in 2022? I think this will be an on-going activity for the first two years. 10% of Goldina, Zhang, Ciardi, Surace 20% of Roby, Rector, Ly, Wu 40% Groom | 100 |
| 485 | DM-1149 | W15 Metadata Store for production tracking (prototype) | Build a first prototype of the Metadata Store. JK: Refer to loading spreadsheet for PMCS assignments | 45 |
| 486 | DM-1150 | Fast image search | A lot of users will search for images using some (often advanced) spatial criteria. Need to implement something similar to what we have in UDFs/SciSQL to efficiently support this class of search. Check GIS support in MariaDB | 10 |
| 487 | DM-1151 | Fix example of IsrTask to be callable with data on disk | Currently the example of the IsrTask takes a fake dataref. This is hard to use with real data. In DM-1113 we will update IsrTask to not take a dataRef. This will make it easy to update the example script to work with real data. This ticket will also include removing from the unit tests any fake dataRefs that have become unnecessary as a result of DM-1299. | 2 |
| 488 | DM-1152 | Css C++ client needs to auto-reconnect | The zookeeper client in C++ that the czar uses doesn't auto-reconnect. This is a capability provided in the kazoo library that qserv's python layer provides, but isn't provided in the c++ client. The zookeeper client disconnects pretty easily: if you step through your code in gdb, the zk client will probably disconnect because its threads expect to keep running. zk sessions may expire too. Our layer should reconnect unless there is really no way to recover without assistance from the calling code (e.g. configuration is wrong, etc.). This ticket includes only basic reconnection attempting, throwing an exception only when some "reconnection-is-impossible" condition is met. | 2 |
| 489 | DM-1153 | Minor problems in lsstsw, related to Qserv offline install procedure | - on lsstsw master branch tip, ./stack/Linux64/lsst/9.2/bin/newinstall.sh doesn't seems to be the last version (it still install eups-1.3.0 instead od eups-1.5.0) - on Fedora19, flock from util-linux 2.23.1 doesn't support next options : {code:bash} $ flock -w 0 200 flock: timeout cannot be zero {code} but {code:bash} flock -w 1 200 {code} works, - newinstall.sh : would it be possible to enable automatic answers to git and anaconda install questions. For example, in order to easily enable automatic install on 300 nodes clusters ? (cancelled : covered by DM-1078) - loadLSST.sh appends automatically http://sw.lsstcorp.org/eupspkg to EUPS_PKGROOT, and if first url in EUPS_PKGROOT isn't available eups fails without trying next ones => this isn't compliant with offline mode and introduce a work-around in Qserv offline mode install scripts. Would it be possible to define a lightly different behaviour for loadLSST.sh ? - In newinstall.sh, l. 167, if Python version isn't correct, then exit *with error code*. | 3 |
| 490 | DM-1154 | SUI web user interface prototype | SUI team want to use this time to study the DM system and all the other design documents to come up with preliminary design of the SUI infrastructure with some prototyping along the way to help the proof of concept. 40% Wu 30% Rector, Roby, Ly 10% Zhang, Goldina 80% Ciardi 90% Surace 60% Groom | 100 |
| 491 | DM-1155 | SUI Interface through Qserv with database and prototype | Working closely with database group to define the interface needs between SUI and Qserv. Prototyping the functions will help us understand the interface better. 50% Rector 10% Ly 10% Wu | 32 |
| 492 | DM-1156 | SUI: Query and display LSST image | Define the APIs for image query with database group. Depend on the implementation of APIs Exercise the image cutout service DM-1977 been developed in SLAC. | 6 |
| 493 | DM-1157 | SUI catalog query interface prototype | Define the APIs for image query with database group Depend on the implementation of APIs 30% Goldina 20% Ly 10% Wu | 34 |
| 494 | DM-1158 | SUI image visualization prototype (without searching LSST images) | Using the current software components developed in IPAC to put together a prototype of visualization capabilities. The purpose is to get feedback from DM people and potential users of the tool. 20% Roby 30% Zhang | 34 |
| 495 | DM-1159 | SUI 2D plot for catalog prototype | Using the current software components developed in IPAC to put together a prototype of visualization capabilities. The purpose is to get feedback from DM people and potential users of the tool. 30% Goldina 10% Ly | 38 |
| 496 | DM-1160 | SUI catalog and image interactive visualization with LSST data | Using the current software components developed in IPAC to put together a prototype of visualization capabilities. The purpose is to exercise the data access APIs developed by SLAC and get feedback from DM people and potential users of the tool. 20% Goldina, Zhang 10% Roby, Ly, Wu, Ciardi | 20 |
| 497 | DM-1161 | Cleanup SdssShape | We should do a comprehensive cleanup of the SdssShapeAlgorithm class. This includes removing the SdssShapeImpl interface (never supposed to have been public, but it became public) from other code that uses it, and integrating this code directly into the algorithm class. We should also ensure that the source from which the algorithm is derived is clearly cited -- that's Bernstein and Jarvis (2002, http://adsabs.harvard.edu/abs/2002AJ....123..583B); see also DM-2304. | 8 |
| 498 | DM-1162 | Port meas_algorithms unit tests for plugins | While test coverage isn't complete, there are many unit tests for measurement plugin algorithms in meas_algorithms that have not yet been ported to meas_base. We should make sure any of these tests that aren't already covered in meas_base are moved over before we remove the meas_algorithms versions of things. | 6 |
| 499 | DM-1164 | remove dependency between sconsUtils and eups | sconsUtils currently depends on eups, (and as far as I understand it, it should not...) | 4 |
| 500 | DM-1167 | experiment with building MyISAM as a shared library | Per Monty (phone call Aug 28, 2014), this should be easy, code to look at is in storage/myisam/mi_test* | 2 |
| 501 | DM-1168 | zookeeper port numbers should be configurable | The test programs in core/modules/css/ has hardcoded zookeeper port numbers. That needs to be fixed, it needs to be configurable. | 1 |
| 502 | DM-1169 | User-friendly install/configure/test scripts | - stdout, stderr output should be colorized, whereas redirecting it into a file shouldn't. | 3 |
| 503 | DM-1170 | FY19 Add Support for ForcedSource Table in Qserv | ForcedSources will needs special handling * ra/decl columns are not part of ForcedSource table, we will need them for partitioning * we might want to store them in subchunks? | 53 |
| 504 | DM-1176 | Make default image origin PARENT in all cases | After DM-840 is finished, and code updated for it (DM-846), make the new default image origin PARENT, remove the UNDEFINED image origin enum, and update at least some of the code that explicitly specifies PARENT (there is little harm in leaving this explicit, other than it does not set an optimal example). | 4 |
| 505 | DM-1177 | Implement sharable just-in-time subchunk management on the worker | The current subchunk management on the worker is very simple. When a query fragment is selected for execution, it builds exactly the subchunks it needs, performs the queries, and destroys the subchunks. This presents a problem when we have concurrent queries that need the same subchunks. Thus an earlier query will destroy its subchunks (some/all of which are needed by another concurrent query) and cause the later queries to fail. The suggested approach is as follows: * Each query, instead of building subchunks on its own, delegates responsibility for subchunk creation/deletion to another class (e.g. SubChunkManager, [but try to find a better name]). * Each query requests a subchunks and releases subchunks using a lock()/unlock() mechanism (in the SubChunkManager API), or wrapped in an interface like boost::lock_guard wraps boost::mutex where acquisition and deletion can be handled by scoping. (It might even be possible to use boost::lock_guard mechanism.) * Now SubChunkManager blocks calls to lock() until it has created the necessary subchunks and modified its records to note that these subchunks are needed by one additional query). A subsequent unlock() call from the client indicates that SCM can decrement its counter and destroy the appropriate subchunks. I (danielw) think the data structure to manage this is straightforward and the code for creation/deletion of subchunks already exists, but it might not be obvious how to route the plumbing so that queries can request/release appropriately, to an instance (probably only one per worker instance) owned by an object somewhere else. Please feel free to use a better or more maintainable approach. | 15 |
| 506 | DM-1185 | Channel was inactive for too long error | There as an issue with the receiveEvent call that can cause an exception "Channel was inactive for too long". The fix for this (according to the ActiveMQ users list and archives) is to either completely disable the inactivity monitor or to increase the inactivity limit to something extremely high. The fix is adding: "wireFormat.maxInactivityDuration=0" to the URL in establishing the connection. There might also be a way of doing this directly in the ActiveMQ broker, but so far I haven't seen anything that would let me specify that. | 2 |
| 507 | DM-1187 | Learn the OCS middleware | OCS will deliver the OCS Middleware software in November 2014. There will be a workshop held at SLAC the week of November 10th. I spoke with K-T and we should be able to attend this remotely, if necessary. We need to spend some time getting familiar with this software and how it will integrate with the Base DMCS for AP. | 6 |
| 508 | DM-1188 | rewrite low-level shapelet evaluation code | While trying to track down some bugs on DM-641, I've grown frustrated with the difficulty of testing the deeply-buried (i.e. interfaces I want to test are private) shapelet evaluation code there. That sort of code really belongs in the shapelet package (not meas_multifit) anyway, where I have a lot of similar code, so on this issue I'm going to move it there and refactor the existing code so it all fits together better. | 2 |
| 509 | DM-1192 | Write a transition plan to move gitolite and Stash repositories to GitHub | As recommended by the SAT meeting on 2014-09-16, we need this document to promote the use of GitHub by other subsystems within the project and to understand the impacts on DM. The plan should include, but is not limited to: * Whether and how the repositories should be reorganized. * How existing commit attributions will be translated. * Moving comments in Stash to GitHub | 20 |
| 510 | DM-1195 | There is a bug in the prescan bbox for megacam. | The bounding box of the prescan region in the megacam camera should have zero y extent (I think). Instead it goes from y=-1 to y=2. This is either a bug in the generation of the ampInfoTables or in the way the bounding boxes are interpreted. | 1 |
| 511 | DM-1196 | exampleUtils in ip_isr is wrong about read corner | https://dev.lsstcorp.org/cgit/LSST/DMS/ip_isr.git/tree/examples/exampleUtils.py#n95 Says that the read corner is in assembled coordinates. This is not true, it is in the coordinates of the raw amp. That is, if the raw amp is in electronic coordinates (like the lsstSim images) it is always LL, but if it is pre-assembled, it may be some other corner. This should probably use the methods in cameraGeom.utils to do the image generation. | 1 |
| 512 | DM-1197 | Support some mixed-type operations for Point and Extent | The current lack of automatic conversions in python is pretty irritating, and I think it's a big enough issue for people writing scripts that we should fix it. In particular, allow {code} Point2D + Extent2I Point2D - Extent2I Point2D - Point2I Extend2D + Extent2I Extend2D - Extent2I {code} (and the respective operations in the opposite order where well defined) It would also be good to allow the all functions expecting PointD to accept PointI, but I'm not sure if swig makes this possible. It's probably not worth providing C++ overloads for all of these functions (and to be consistent we should probably do all or none). I realize that you invented these types to avoid bare 2-tuples, but I'm not convinced that we shouldn't also provide overloads to transparently convert tuples to afwGeom objects. | 2 |
| 513 | DM-1199 | Revisit log integration with Xrootd | The integration of logging with xrootd needs to be revisited, for both the czar and the worker, after a discussion with Xrootd devs about API changes. We want to accomplish 2 things: * Logging from xrootd client should go through the qserv-czar's logger, so it can be saved in the same file and enabled/disabled as a component. * Logging from the xrootd server may also find a benefit in using our logger as a backend | 20 |
| 514 | DM-1201 | add ColumnView support to FunctorKeys | FunctorKeys currently only work with individual records, but at least some could work with columns as well. Need to add an interface for this and decide how to handle cases where it the FunctorKey doesn't support it. | 2 |
| 515 | DM-1210 | Invert buffering for czar in row-based result handling | The XrdSsi API performs some impedance-matching in buffering transfers. The current code (introduced in DM-199) doesn't leverage this because its flow model is based on the older mysqldump-based results passing. With dump files, we don't know the size of the fragments expected, so we are just passing buffers of bytes and building up a text blob to ingest. With the row-based protocol, we have sizes specified, hence the receiver can specify exactly what sizes of bytes are needed. Implementation of this ticket should simplify and reduce the code overall. | 10 |
| 516 | DM-1211 | anaconda is too outdated to work with pip | The version of anaconda distributed with the stack is too outdated to be used with pip (and probably other things). The issue is an unsafe version of ssh. A workaround is to issue this command while anaconda is setup: {code} conda update conda {code} Warning: it is unwise to try to update anaconda itself (with "conda update anaconda") because that will revert some of the changes and may result in an unusable anaconda. I think what is required is an obvious change to ups/eupspkg.cfg.sh The current version of anaconda is 2.0.1 based on http://repo.continuum.io/archive/ Note: there is no component for anaconda. I will submit another ticket. | 2 |
| 517 | DM-1213 | cleanup order/grouping of header files | We want: * header for the class * then system * then third party * then lsst * then qserv We currently don't have the "lsst" group (with a few exceptions), and we call the last one "local" in most places. | 1 |
| 518 | DM-1215 | makeMaskedImage leaks memory | Calling afw.image.makeMaskedImage leaks memory when called from Python, because it returns a raw pointer without telling Swig %newobject. To fix it, it'd be better to just have it return by value instead of by pointer, though this might involve fixing some downstream C\+\+ code (Python code should not be affected). | 1 |
| 519 | DM-1216 | compute linear parameter derivatives more intelligently in optimizer | The numerical derivatives computed by the optimizer currently don't distinguish between the linear parameters (for which derivatives are trivial) and nonlinear parameters (for which they're hard), because we don't pass the information that distinguishes them to the object that computes the derivatives. If we move the computation of derivatives from the Optimizer class to the Objective class, we should be able to compute the derivatives much more efficiently. While this doesn't matter much when fitting single component galaxy models (because there's only one linear parameter in that case), it should matter quite a bit when fitting high-order shapelets to PSF models. | 4 |
| 520 | DM-1217 | Refactor meas_base Python wrappers and plugin registration | meas_base currently has a single Swig library (like most packages), defined within a single .i file (like some packages). It also registers all of its plugins in a single python module, plugins.py. Instead, it should: - Have two Swig libraries: one for the interfaces and helper classes, and one for plugin algorithms. Most downstream packages will only want to %import (and hence #include) the interface, and having them build against everything slows the build down unnecessarily. The package __init__.py should import all symbols from both libraries, so the change would be transparent to the user. - Have separate .i files for each algorithm or small group of algorithms. Each of these could %import the interface library file and the pure-Python registry code, and then register the plugins wrapped there within a %pythoncode block. That'd make the implementation of the algorithms a bit less scattered throughout the package, making them easier to maintain and better examples for new plugins. | 3 |
| 521 | DM-1218 | Support multiple-aperture fluxes in slots | We should be able to use multiple-aperture flux results in slots. While this is technically possible already by setting specific aliases, it doesn't work through the usual mechanisms for setting up slots (the define methods in SourceTable and the SourceSlotConfig in meas_base). After addressing this, we should remove the old SincFlux and NaiveFlux algorithms, as the new CircularApertureFlux algorithm will be able to do everything they can do. | 2 |
| 522 | DM-1226 | Refine Butler prototype | A prototype of the new Butler is part of the S14 delivery. This needs to be refined into a production package and the rest of the code needs to be ported to use it. | 15 |
| 523 | DM-1227 | S15 Multi-node Multi-query Integration Testing Harness | Build end-to-end integration test harness for Qserv that will run queries on multi-node system. | 56 |
| 524 | DM-1228 | LSE-68: Bring pull interface to CCB approval | Bring a long-pending set of changes, primarily the adoption of the "pull interface" for all DM-Camera image requests, to CCB approval | 6 |
| 525 | DM-1229 | LSE-68: Major phase-3 details (W15) | Deepen ISD to cover: supply of crosstalk constants, deletion policy details, content of the new-data notification, availability of a pass-through tag in data, and other Phase 3 matters. Edit ISD to ensure that it covers WFS and guider requirements as well. Deliverables: * Marked-up LSE-68 with combination of DM proposals for Phase 3 details and, where that is not a realistic approach, specific questions for Camera DAQ team. * Bring a proposal to the CCB by the end of the cycle. | 16 |
| 526 | DM-1230 | LSE-69: Bring Summer 2014 work to CCB approval | Bring Summer 2014 work on LSE-69 to CCB approval, ideally by 20 October in time to be part of the CD-2 document package. | 6 |
| 527 | DM-1232 | LSE-72: Bring Summer 2014 work to CCB approval | Remaining work is to proofread the SysML-ization by Brian Selvy of the LSE-72 draft, do any required cleanup in conjunction with the OCS team, and advocate for LCR-202 (already exists) at the CCB. | 3 |
| 528 | DM-1233 | Refine requirements and use cases for Level 3 facilities | Refine the requirements and use cases for the three branches of Level 3 capabilities exposed to users: * Level 3 programming toolkit (user reconfiguration / extension of DM pipelines and stack) * Level 3 compute cycle delivery (user access to 10% of compute base) * Level 3 data product storage Deliverables: * Refinement, if necessary, to Level 3 requirements in DMSR * Flowed-down requirements as a separate document. Sufficient detail to allow a breakdown of the deliverables in the three areas of Level 3 by annual release cycle through construction period. | 20 |
| 529 | DM-1234 | LSE-72: Remaining Phase 2 details | Sort out remaining Phase 2 details of LSE-72, especially: * details of the configuration mechanism (yet to appear even in LSE-70), the publication of available configuration keys, the efficient publication of configuration contents, OCS-driven configuration "knobbing" * specific list of events to be published by DM * more specific EFD-query language * EFD deployment model (understanding of distributed design envisioned by OCS and its implications for the ICD) | 10 |
| 530 | DM-1235 | LSE-72: Phase 3 details | Advance Phase 3 details as needed to eliminate obstacles to OCS and DM development during Summer 15. | 10 |
| 531 | DM-1236 | LSE-75: Bring Summer 2014 work to CCB | Bring a small set of technical changes (e.g., pointing notice moved to OCS ICD) and the addition of PSF reporting to T&S into the SysML version of LSE-75 and submit a change request to get this approved. | 6 |
| 532 | DM-1237 | LSE-75: Refine WCS and PSF requirements | Clarify the data format and precision requirements of the TCS (or other Telescope and Site components) on the reporting of WCS and PSF information by DM on a per-image basis. Depends on the ability of the T&S group to engage with this subject during the Winter 2015 period. Can be deferred to Summer 2015 without major impacts. Current PMCS deadline for Phase 3 readiness of LSE-75 is 29-Sep-2015. | 8 |
| 533 | DM-1238 | LSE-130: Bring Summer 2014 work to CCB approval | Bring the current list-oriented version of LSE-130 into the camera-test-plan-oriented format requested by the Camera group, moving the old list(s) into LDM-272 for reference. Negotiate what further work is required to make the document acceptable to the Camera group for CCB approval in time for CD-2. Nominally this requires CCB approval by 20-Oct-2014 in order to have an approved document properly released to the CD-2 committee. | 16 |
| 534 | DM-1239 | LSE-130: Make requests more quantitative | Proceed, as enabled by input from the DM Apps group and the Camera, to make the individual data requests in LSE-130 more quantitative. | 12 |
| 535 | DM-1240 | LSE-140: Bring Summer 2014 work to CCB approval | Convert the existing LSE-140 draft to SysML, produce a docgen, and review with Jacques Sebag. Bring to CCB meeting on 8 October under existing LCR-201. | 6 |
| 536 | DM-1241 | Complete data entry of LSE-140 revised draft into EA | NULL | 2 |
| 537 | DM-1242 | Risk Register refresh 1/2015 | Periodic review of DM risk register contents. Covers preparation for a review expected at the end of January 2015, the only one during Winter 2015. | 3 |
| 538 | DM-1244 | TOWG - Contributions to the Operations Concept Document | Covers contributions to the writing and editing of the Operations Concept Document during the Winter 2015 cycle. Deliverables: * Timescales diagram and explanatory text * Framework for Nominal Operations chapter * Contributions to KTL's writing of the Data Processing Operations chapter * General proofreading | 10 |
| 539 | DM-1245 | Install scisql plugin (shared library) outside of eups stack. | sciSQL plugin is currently deployed in eups stack (i.e. $MYSQL_DIR/lib/plugin) during configuration step. Nevertheless eups stack should be immutable during configuration step. MySQL plugin-dir option may allow to deploy sciSQL plugin outside of eups stack (for example in QSERV_RUN_DIR). | 3 |
| 540 | DM-1246 | add per-exposure callbacks for measurement plugins | We should give measurement plugins an opportunity to do some work whenever we start processing a new Exposure. This give them a good time to throw exceptions when there's something obviously wrong with the exposure (e.g. it's missing a Psf). It also gives them an opportunity to do work that could be used to speed up per-source processing. One specific case I have in mind is in shapelet PSF approximation for galaxy fitting, where it could be very valuable to use an initial fit to an average PSF to initialize (and speed up) the first to per-source PSFs. One question here is how to to allow information to be passed from the per-Exposure method to the per-Source methods - we should not do that via plugin instance attributes, which means we probably want to have the per-Exposure method return an arbitrary object that would be passed to the per-Source methods. Unfortunately, I don't see a way to avoid having that change the signature for those methods for all existing plugins, so this should be done relatively early, before we have too many user-contributed plugins. | 3 |
| 541 | DM-1249 | Reimplement CSS using JSON-packing, czar kazoo + c++ snapshotting phase 1 | Transient cache for CSS, populated from python, no C++ interface to zookeeper. Pack multiple keys using json. Tenative packing spec is in DM-705 The v1 implementation transitions the zk access from the c++ layer into python, with code that understands how to unpack json-encoded data. This can coexist with the existing qserv_admin zk schema. v2 applies packing/unpacking logic in the creation/manipulation code in qserv_admin. | 10 |
| 542 | DM-1250 | CSS design for query metadata v2 | NULL | 3 |
| 543 | DM-1251 | CSS design for query metadata v1 | The goal of this ticket (and DM-1250) is to try to understand what kind of per-query metadata is necessary to provide client-transparent query processing in case when czar/proxy could die or be restarted. Some relevant info is in the Trac: https://dev.lsstcorp.org/trac/wiki/db/Qserv/CSS/RunTimeState | 3 |
| 544 | DM-1252 | Implement structure for DB/table metadata in CSS | NULL | 4 |
| 545 | DM-1253 | Requirements gathering for Metadata Store | NULL | 10 |
| 546 | DM-1254 | Metadata Store - design v1 | Research potential off-the-shelf candidates. Propose initial version of metadata design. | 5 |
| 547 | DM-1255 | Metadata Store - experimental prototype v1 | This first prototype will involve capturing metadata from a small set of image files. * Create a new schema file in the cat package (lsstSchema4mysqlW15.sql) with definition of tables that will be used to capture image metadata (see Exposure tables in other schema files), and some basic structure for capturing information about the entire repository. Decide which keywords are standard, and which should go to flat key-value area * Pick a directory (or several) with images, candidates: /lsst7/dr7/runs, /lsst7/releaseW13EP, /lsst3/DC3/data/obs/ImSim/pt1_2, /lsst/DC3/data/pt1.2.3k/datarel/ImSim * create test database, load schema * extract metadata for all images in a given directory and load metadata from fits headers. | 10 |
| 548 | DM-1256 | Metadata Store - design v2 | NULL | 5 |
| 549 | DM-1257 | Metadata Store - experimental prototype v2 (DataCat) | Integrate prototype v1 with Fermi's DataCat (e.g., reuse logic for reading headers using afw, store in DataCat). Experiment with foreign tables. | 6 |
| 550 | DM-1258 | Update documentation and automatic install script w.r.t. Qserv 2014_09.0 release | Creation of qserv_distrib and distribution of Qserv via official LSST repositories have to be taken into account in Qserv documentation and automatic install script. | 4 |
| 551 | DM-1259 | evaluation of Cinder as OpenStack storage cache in the LSST Middleware | Deliverable: evaluation of Cinder as OpenStack storage cache in the LSST Middleware Daues G 100%" | 100 |
| 552 | DM-1260 | implementation of data movement in the production system in the existing hierarchy of NFS disk, condo, tape | Deliverable: implementation of data movement in the production system in the existing hierarchy of NFS disk, condo, tape Freemon M 50% | 54 |
| 553 | DM-1261 | incorporation of sizing model into data center requirements | Deliverable: incorporation of sizing model into data center requirements Freemon M 100% | 7 |
| 554 | DM-1262 | data center requirements document | Deliverable: data center requirements document Petravick D 10% | 11 |
| 555 | DM-1263 | completed governance of security plan for review | Deliverable: completed governance of security plan for review Petravick D 10% | 1 |
| 556 | DM-1264 | security plan october. | Deliverable: security plan Ephibian 10% | 1 |
| 557 | DM-1265 | addition of procurement of physical goods to contract | Deliverable: addition of procurement of physical goods to contract Petravick D 5%, Gelman M 10% | 6 |
| 558 | DM-1266 | insertion of wide area simulator into test stand | Deliverable: insertion of wide area simulator into test stand Freemon M 100% | 18 |
| 559 | DM-1267 | report of traffic shaping | Deliverable: report of traffic shaping Freemon M 100% | 7 |
| 560 | DM-1268 | upgraded KVMs | Deliverable: upgraded KVMs Mather B 50% | 9 |
| 561 | DM-1269 | sizing model critique report | Deliverable: sizing model critique report Perez A 50% | 29 |
| 562 | DM-1270 | more efficient VMware infrastructure | Deliverable: more efficient VMware infrastructure W15: Glick B 50%, Elliott M 25%, Mather B 10% 38 SP estimated S15: Mather B 40%, Glick B 25% 6 SP estimated | 6 |
| 563 | DM-1271 | technical roadmap critique report | Deliverable: technical roadmap critique report Perez A 50% | 29 |
| 564 | DM-1272 | Upgrade NFS Storage Servers with new hardware | Deliverable: new hardware, ZFS filesystem, 3x 100+ TB + 1 spare 100+ TB Elliot M 75% | 55 |
| 565 | DM-1273 | deployment plan for version 1 of OpenStack | Deliverable: deployment plan for version 1 of OpenStack Glick B 75%, Elliot M 15%, Mather B 10%, Wefel P 10% | 56 |
| 566 | DM-1274 | Procure replacement development infrastructure | Procure, install, test, and deploy hardware to replace existing LSST development cluster infrastructure. Assignees: Bill Glick, Matt Elliot, Bruce Mather, Paul Wefel, Jason Alt Duration: November - December 2015 | 45 |
| 567 | DM-1275 | level of effort | Deliverable: level of effort Wefel P 100%, Freemon M 5% | 100 |
| 568 | DM-1276 | level of effort | Deliverable: level of effort Voiciu L 100% | 28 |
| 569 | DM-1277 | level of effort | Deliverable: level of effort Petravick D 50%, Gelman M 50%, Glick B 30%, Mather B 50% | 100 |
| 570 | DM-1278 | written plan for next period epics | Deliverable: written plan for next period epics Petravick D 50%, Gelman M 50%, Glick B 50% | 10 |
| 571 | DM-1279 | Fix Scisql deployment test error (doc.py) | Deployment test in tools/docs.py fails due to a wrong "scisql_index" path in scisql documentation. Fortunately, qserv-configure.py doesn't stop on this error. | 2 |
| 572 | DM-1280 | meas_base ResultMappers should be FunctorKeys | The ResultMapper classes in meas_base should inherit from FunctorKey, and support bidirectional transfers involving the Result structs and records. | 3 |
| 573 | DM-1281 | add Schema method to join strings using the appropriate delimiter | Delimiters in Schema field names are version-dependent. One can currently use {{schema["a"]["b"].getPrefix()}} to join fields using the appropriate delimiter, but this is confusing to read. | 1 |
| 574 | DM-1282 | multi-level replacement in Schema aliases | Schema aliases should support more than one level (i.e. an alias may resolve to another alias). | 2 |
| 575 | DM-1283 | remove meas_extensions_multiShapelet from release packages/buildbot | NULL | 1 |
| 576 | DM-1284 | rename meas_multifit to meas_modelfit, and add to lsst_apps | NULL | 1 |
| 577 | DM-1285 | Improve Startup of HTCondor Jobs | Adjust configuration parameters of HTCondor config and/or submission files to improve speed at which HTCondor jobs start in both the replicator pool and worker pool. | 2 |
| 578 | DM-1286 | Improve worker fault tolerance of missing distributor data | Instances of worker jobs might ask the Archive DMCS which distributor to connect to, only to connect to that distributor which has recently been rebooted, so the distributor might not have that information. At this point, the Distributor could send an “expired” notice of some kind to the Archive DMCS to clear it’s cache, and also tell the worker that it has no file of the type it’s looking for. The worker would then go back to the Archive DMCS. | 6 |
| 579 | DM-1287 | Propose and document a recipe to build Qserv in eups | In-place build is available and documented. | 2 |
| 580 | DM-1291 | Identify test data and camera | To test processCcd we need to identify a set of data to process (possibly mocked). This implies that there will also need to be a minimal camera to go with the minimal data. This task is to identify the minimal data and find a location for it. | 4 |
| 581 | DM-1293 | Implement designed tests for processCcd | Implement the designed tests with the installed data. | 8 |
| 582 | DM-1294 | Generate use case | We should first sit down with a Camera team rep (Jim C. maybe) to define the tool they need. How dynamic are the data? What is the layout of the data? We should also get some example test data to work with. | 6 |
| 583 | DM-1295 | Sync test data headers with standards | The headers will need to be vetted against known FITS standards. The headers should be valid and should not contain any non-standard keywords. There should also be a census of FITS standards that may be used for defining sensor layout. | 8 |
| 584 | DM-1296 | Implement a tool to generate a Camera from FITS images | Implement the tool to generate the camera object. It should be a command line tool that will take a path to a file. It should have an option to persist the generated camera. It should also have the ability to plot the camera. | 10 |
| 585 | DM-1297 | Verify use of the tool with the camera team | We will need to run the tool in the camera team's system to make sure the interfaces are as they expected. It will also help to make sure that the tool addresses all of the cases. | 6 |
| 586 | DM-1298 | Design the API | The API will have to be able to accommodate data we know about, so will need to deal with reasonable missing data. It should also not preclude extension. This will need to be RFC'd since it is an API change. | 8 |
| 587 | DM-1299 | Implement and test the new API | Once the API is designed and signed off on, the API will need to be implemented and tested. This will require updating all obs_* packages that use the current interface. | 10 |
| 588 | DM-1302 | C++ code changes required for --std=c++11 | Some C++ code requires changes for modern C++ compilers if it is to be compatible with C++11 and the older standard. Here is my list, so far (ignoring the known issue of MacOS having two different standard C++ libraries) Implicit conversion of shared_ptr to bool no longer works: <http://stackoverflow.com/questions/7580009/gcc-error-cannot-convert-const-shared-ptr-to-bool-in-return> In C++11, shared_ptr has an explicit operator bool which means that a shared_ptr can't be implicitly converted to a bool. This is likely to show up in a lot of packages. So far found in: - pex_policy - meas_algorithms warning: adding 'int' to a string does not append to the string seen in daf_persistence warning: 'va_start' has undefined behavior with reference types seen in pex_logging Trace.h and one other place. Fixed by not using references (and thus copying the arguments), rather than using pointers, to avoid changing the APIs. this is an old issue; but not all compilers warned about it: See <http://stackoverflow.com/questions/222195/are-there-gotchas-using-varargs-with-reference-parameters> warning: 'register' storage class specifier is deprecated seen in: - boost 1.55.0.1 - Eigen 3.2.0 - a pex_logging .i file I silenced this warning in sconsUtils because it's too much trouble to upgrade boost and Eigen, and the warnings are very obtrusive in daf_persistence: {code} src/DbStorageImpl.cc:87:35: warning: first declaration of static data member specialization of 'mysqlType' outside namespace 'persistence' is a C++11 extension [-Wc++11-extensions] dafPer::IntegerTypeTraits<1>::mysqlType = MYSQL_TYPE_TINY; {code} I have not figured out how to fix this one. clang 6 is pickier about the order of instantiation: in afw (fixed in DM-1302) Many errors such as the following: {code} include/lsst/afw/table/Flag.h:27:8: error: explicit specialization of 'lsst::afw::table::FieldBase<lsst::afw::table::Flag>' after instantiation struct FieldBase<Flag> { ^~~~~~~~~~~~~~~ include/lsst/afw/table/Key.h:49:39: note: implicit instantiation first required here {code} Fixed by including Flag.h in Key.h clang 6 warns about ambiguous unsequenced modifications. Seen in shapelet, for example: {code} src/HermiteTransformMatrix.cc:107:59: warning: multiple unsequenced modifications to 'kn' [-Wunsequenced] for (int kn=jn, koff=joff; kn <= order; (koff += (++kn)) += (++kn)) { ^ ~~ {code} Fixed by using a += ++b, a += ++b, since it was the cleanest solution I could find. clang 6 is pickier about instantiation classes in the wrong namespace: in shapelet (fixed by not instantiating the anonymous classes): {code} src/MatrixBuilder.cc:945:1: error: explicit instantiation of 'lsst::shapelet::<anonymous namespace>::SimpleImpl' must occur in namespace '' INSTANTIATE(float); {code} meas_algorithms produced this warning because the class member _wcsPtr was a reference (which was unecessary): {code} src/ShapeletKernel.cc:188:34: warning: binding reference member '_wcsPtr' to a temporary value [-Wdangling-field] _interp(interp), _wcsPtr(wcsPtr->clone()) ^~~~~~~~~~~~~~~ {code} clang 6 warns about expressions such as if (!a == 0) because the ! is applied to "a", not the result of the "a == 0". Fixed in the obvious way in several places. meas_algorithms seg-faults on loading unless boost is built with C++11 support. See ticket DM-1361 | 4 |
| 589 | DM-1305 | Tests fail in shapelet when building on OS X 10.9 | When building the master on pugsley.ncsa.illinois.edu, shapelet builds successfully, but two tests fail: {code} pugsley:lsstsw mjuric$ cat build/shapelet/tests/.tests/*.failed tests/testMatrixBuilder.py .F..... ====================================================================== FAIL: testConvolvedCompoundMatrixBuilder (__main__.MatrixBuilderTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "tests/testMatrixBuilder.py", line 310, in testConvolvedCompoundMatrixBuilder self.assertClose(numpy.dot(matrix1D, coefficients), checkVector, rtol=1E-14) File "/Users/mjuric/test/lsstsw/stack/DarwinX86/utils/9.2+8/python/lsst/utils/tests.py", line 328, in assertClose testCase.assertFalse(failed, msg="\n".join(msg)) AssertionError: 1/50 elements differ with rtol=1e-14, atol=2.22044604925e-16 0.175869366369 != 0.175869366369 (diff=1.99840144433e-15/0.175869366369=1.13629876856e-14) ---------------------------------------------------------------------- Ran 7 tests in 0.323s FAILED (failures=1) tests/testMultiShapelet.py ...F... ====================================================================== FAIL: testConvolveGaussians (__main__.MultiShapeletTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "tests/testMultiShapelet.py", line 88, in testConvolveGaussians self.compareMultiShapeletFunctions(msf3a, msf3b) File "/Users/mjuric/test/lsstsw/build/shapelet/python/lsst/shapelet/tests.py", line 107, in compareMultiShapeletFunctions self.compareShapeletFunctions(sa, sb, rtolEllipse=rtolEllipse, rtolCoeff=rtolCoeff) File "/Users/mjuric/test/lsstsw/build/shapelet/python/lsst/shapelet/tests.py", line 86, in compareShapeletFunctions rtol=rtolEllipse) File "/Users/mjuric/test/lsstsw/stack/DarwinX86/utils/9.2+8/python/lsst/utils/tests.py", line 328, in assertClose testCase.assertFalse(failed, msg="\n".join(msg)) AssertionError: 1/5 elements differ with rtol=1e-14, atol=2.22044604925e-16 2.44929359829e-16 != 1.13310777953e-15 (diff=8.881784197e-16/1.13310777953e-15=0.783842839795) ---------------------------------------------------------------------- Ran 7 tests in 0.131s FAILED (failures=1) {code} =============== More info on pugsley.ncsa.illinois.edu: pugsley:lsstsw mjuric$ sw_vers ProductName: Mac OS X ProductVersion: 10.9.5 BuildVersion: 13F34 pugsley:lsstsw mjuric$ clang -v Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn) Target: x86_64-apple-darwin13.4.0 Thread model: posix ============ The files are in {{/Users/mjuric/test/lsstsw/build/shapelet/}}. | 1 |
| 590 | DM-1306 | Pre-CCB review of LSE-140 docgen | NULL | 2 |
| 591 | DM-1309 | Edit agreed-upon changes into Word version of LSE-69 | A meeting around 9/26/2014 agreed on a set of revisions to LSE-69, with some language still needed from [~gpdf]. This action is to edit the tracked-changes Word version of LSE-69 containing the notes from that meeting into a final copy that can be reviewed by the Camera team and used as input to editing the SysML version of the ICD. | 3 |
| 592 | DM-1310 | Create change request for LSE-69 | Create a change request to bring LSE-69 up to date and capture the Summer 2014 work. | 1 |
| 593 | DM-1311 | Enter LSE-69 update into EA as SysML | Covers entering the contents of the LSE-69 update into EA as SysML, with associated updating of diagrams, and the creation of a docgen'ed version for CCB action. | 1 |
| 594 | DM-1312 | Proofread docgen'ed version of LSE-72 | Brian Selvy is producing a SysML version of the LSE-72 updated edited by [~gpdf]. The action here is to proofread the docgen of that version once it is ready. | 2 |
| 595 | DM-1313 | Identify Conditions information in LSE-130 that is required for Alert Production | LSE-69 declares that there are two categories of Conditions data (telemetry) required by DM from the Camera: those items that are needed for Alert Production (for which the AP components at the Base will need a whitelist, and for which the Camera has a tighter latency requirement), and those that are not (but are then presumably needed in DRP or other deferred productions). It states that the subset needed for AP should be enumerated in LSE-130. The action here is to create an initial version of that list. | 2 |
| 596 | DM-1314 | Publish Qserv S14 version on lsst distribution server | In order to publish this version please tag Qserv master tip with "2014_09.0" and then run: {code:bash} ssh lsstsw@lsst-dev # command below can't be runned in buildbot, as it doesn't support qserv_distrib build rebuild -t 2014_09.0 qserv_distrib # bXXX is provided by previous command publish -t qserv -b bXXX qserv_distrib publish -t 2014_09 -b bXXX qserv_distrib {code} | 1 |
| 597 | DM-1315 | advance to assigning tier-2 and tier-3 reliability levels | Accommodated Ron Lambert's input on networking equipment. Assigned tier-2 and tier-3 levels to processing systems. | 4 |
| 598 | DM-1316 | Deploy LSST stack within OpenStack instances on ISL testbed | Deploy the LSST Stack within OpenStack instances within the ISL testbed -- this could be for multiple flavors CentOS, Ubuntu, etc, and this could be done by pulling Docker Images to the instances. There will also likely be some initial debugging of starting instances within the ISL platform as a new installation has been stood up Sept 2014. | 1 |
| 599 | DM-1317 | Create Docker Image / Dockerfile for LSST Stack for ubuntu | Create an installation of the LSST Stack v9_2 within a Docker Image for ubuntu for easing the import of LSST software into an OpenStack instance, We create the image utilizing a Dockerfile to make systematic the creation of such images. | 2 |
| 600 | DM-1318 | update expected results file in SDSS demo test | Update the expected outputs in the SDSS DM stack demo repo to match what we expect from the new meas_base framework. | 1 |
| 601 | DM-1319 | Prepare quotes to order new VMware hardware & licenses | NULL | 2 |
| 602 | DM-1320 | Setup temporary NFS datastore for VMs | We expect the primary datastores will be local to the VM compute node, but this NFS datastore will allow us to live migrate a handful of VMs between compute nodes. We may also use this or another datastore as place to create backup snapshots of particular VMs. | 2 |
| 603 | DM-1321 | Setup new VM compute nodes | Once the new VM compute nodes arrive, the hardware needs to be installed, ESXi installed, and networked for use by vSphere. | 6 |
| 604 | DM-1322 | Expire Workers that receive no files | In instances where the files the worker expected to get never arrive, there should be a way of the worker to recognize this (say, a timeout after waiting for the Archive DMCS for some time), and exit. | 4 |
| 605 | DM-1323 | Configure and test new VM infrastructure | This would be configuring to work vSphere, setting up permissions and groups, adding datastores, and testing the use of the new setup. | 10 |
| 606 | DM-1326 | Automatic expiration of replicator jobs | Replicator jobs that receive no data specifically for the visit, raft, and exposure sequence ID within a certain amount of time should self expire to prevent future jobs from running. If this is not done, jobs will back up in the HTCondor queue. | 4 |
| 607 | DM-1328 | Order new NFS servers | We are waiting on the UIUC Business Offices to provide us a new account number for ordering these servers. As soon as we get that account number we need to order the hardware from KOI Computing. | 6 |
| 608 | DM-1329 | Plan new NFS mounts & data organization | This is a task of deciding the new client mounts for the 3 new NFS servers. These should match up with the new storage hierarchy and classification <https://wiki.ncsa.illinois.edu/display/LSST/Storage+Structure>. | 1 |
| 609 | DM-1330 | Install new NFS servers | NULL | 10 |
| 610 | DM-1331 | squash edge errors in SdssCentroid | SdssCentroid doesn't trap exceptions that are thrown due to being too close to the edge, resulting in noisy warnings in the logs. Instead, it should catch the low-level exception and re-throw as MeasurementError, after defining a flag field for this specific failure mode. | 1 |
| 611 | DM-1332 | address no-shape warnings in GaussianFlux | GaussianFlux relies on the shape slot, and puts noisy warnings in the logs when the shape slot fails. However, we probably don't want to add a new flag for GaussianFlux to indicate this failure mode, because it'd be entirely redundant with the shape slot flag. We should figure out some other way to squash this warning - how we do that may depend on whether this is addressed before or after the C++ redesign. We should also consider having GaussianFlux add an alias to the schema to point back at the shape slot flag, creating what looks like a specific flag for this failure while actually just being a link back to the shape slot flag. That's probably not worth doing within the current C++ interface, however, as it'd require some unpleasant mucking around with ResultMappers. | 2 |
| 612 | DM-1333 | resolve factor of two difference in GaussianFlux | After changing the implementation of GaussianFlux to use the shape slot rather than estimate the shape itself by re-running the SdssShape code, Perry saw a 5-15% difference in the fluxes (I'm not sure of the sign). The new behavior (using the shape) is consistent with what we'd have gotten with the old code when the little-used "fixed" config option was enabled (not surprising, as that just looked up the SdssShape measurement by name, instead of via slots). I suspect the difference is coming in because of the factor of two between SdssShape's "raw" measurements - the actual Gaussian-weighted moments - and the factor of 2 it applies to make its measurements equivalent to ideal unweighted moments. The correct weight function to use for GaussianFlux includes this factor of 2 (i.e. it's larger than the "raw" moments), and it's likely either the old code wasn't including this or the new code isn't. We need to determine which one, and if necessary, fix the new code. | 2 |
| 613 | DM-1334 | Test the creation of basic OpenStack instances on the new ISL testbed [IceHouse] | A new version & implementation of the ISL OpenStack testbed is up and running. The new cloud is using IceHouse, the ninth OpenStack release. We get started on this platform by verifying that basic instance creation is working. We target the creation of an instance through the (Horizon) GUI interface, and via the nova CLI. | 1 |
| 614 | DM-1335 | Create instance with a Floating IP Associated through the nova CLI | We see that in working with the Horizon GUI, it is fairly straightforward to give an instance a public IP address by associating a Floating IP with the current local IP. However, we will want to be able to accomplish this task both remotely and programmatically within workflow. As a step towards this, we target the solution of this via the nova CLI. | 2 |
| 615 | DM-1336 | Review advance doc to review with the data center working group | Consult with M. Freemon on the sizing model given the generic description of the computing needed tier-3 and tier-2 support. Make a pass through the underlying data center standards doc for things "any competent designer should know" for example wall plugs on separate circuits form computer circuits. add that to the document, and call meeting of working group for review in late Oct. | 4 |
| 616 | DM-1337 | Security Plan advancement for October | get a draft Project Office detail plan consistent with the level of devleopment of the master plan. (ephiphian). Draft a data classification plan, refer to plan and classes of data in drat materials. Obtain some central place in LSST documentation framework to hold materials being drafted (if supported by the LSST system) Recieve feedback based on portion of plan submitted to LSST project office. Incorporate feedback. (if minor) re-plan (if major) | 18 |
| 617 | DM-1338 | Make QSERV_RUN_DIR scripts able to detect qserv install paths using eups | Ticket related to Mario email (subject: [QSERV-L] Some points/actions from the discussion today) : Making your "qserv data" directory independent of where qserv is installed I think this is a big one, and largely independent of EUPS. You have a problem where you want to use one set of test data potentially with different qserv binaries (not at the same time, of course). I'd argue you should refactor the scripts generated by qserv-configure to either: * get _all_ their information about various paths from a _single_ file, for example, from etc/paths.cfg.sh. Then you can easily regenerate just that file when you need to switch to a different qserv (or zookeper, or what not), or... * refactor the generated scripts to learn from the environment which binaries to run. I.e., if $QSERV_DIR is defined, use that qserv, etc. This will let you switch binaries by simply setup-ing the new one with EUPS. The two are not mutually exclusive -- e.g., all of this logic could be in etc/paths.cfg.sh, and depending on whether this is a development build or a "non-EUPS" build, it can either pick up the paths from the environment or hardcode them. Assuming you did that, your development loop may look something like this: {code} # assuming that qserv-configure.py has already been run # in ../qserv-run # do something with qserv-a clone cd qserv-a setup -r . ... do some edits ... scons # now do the tests cd ../qserv-run ./bin/qserv-start.sh ... do tests ... ./bin/qserv-stop.sh # now switch to qserv-b clone cd ../qserv-b setup -r . # and do the tests again cd ../qserv-run ./bin/qserv-start.sh ... do tests ... ./bin/qserv-stop.sh {code} that is, as qserv-start picks up the relevant products from the environment, there's no need to rebuild/reconfigure the qserv-rundirectory each time. | 7 |
| 618 | DM-1340 | Read through log4cxx documentation and log.git code | Read through the log4cxx documentation and become familiar with how the log.git package is set up. | 2 |
| 619 | DM-1341 | Write/configure tests with existing configurations and appenders | In order to be come more familiar with how to use the log.git package, write some tests to see how existing configurations and appenders are used by the log.git package. | 4 |
| 620 | DM-1342 | Write DM message appender class | Write DM message appender class to be used with log.git package. This might entail writing a configurator class as well; that depends on the investigation of how configurations/appenders are used. | 16 |
| 621 | DM-1343 | Write Unit test for new DM message appender class | Write unit tests for DM message appender class. This might also require some tests for a configurator class, if that class is created. | 2 |
| 622 | DM-1344 | Define gather/scatter mechanism for tasks with generic API | Define gather/scatter mechanism for tasks with generic API. This will use event services (DM Messages) initially, but we would like to support MPI or other communication mechanisms as well. This will involve consulting with Paul about how the existing code is structured. | 8 |
| 623 | DM-1345 | OCS Middleware workshop | Attend OCS Middleware workshop. This will probably have be done remotely because I have a personal conflict with that time that will prevent me from attending in person. | 4 |
| 624 | DM-1346 | Write example programs for OCS Middleware | Write some example programs to get familiar with the OCS middleware software. The OCS middleware will later be integrated with the AP, in the base dmcs and replicator jobs. | 4 |
| 625 | DM-1347 | Refine Event base class to allow ActiveMQ filterable settings | The current Event.cc base class needs to be refined to remove and old-style data release terms that aren't used anymore. Plus, it needs to be easily extensible to allow other types of dictionaries of terms that will be used in the message headers to make them filterable on the server side. | 2 |
| 626 | DM-1348 | Update tests to use unit test framework | The tests for this package predate the unit test framework that other package use. Update the tests to uses the unit test framework and get rid of any duplicate or obsolete tests. | 5 |
| 627 | DM-1349 | Change marshaling code to use json | The marshaling code for non-standard (i.e., non-filterable) components of messages is custom and not standard. Change this to use JSON. | 10 |
| 628 | DM-1350 | General cleanup of Events package | There are some obsolete classes and code in the ctrl_events package that needs to be removed and/or updated. PipelineLogEvent, for example. That not only is no longer used, but it is applications specific, and should have been part of another package in the first place, subclassed from this package. | 35 |
| 629 | DM-1352 | ORIGINATORID value can churn too quickly. | The ORIGINATORID is a 64-bit word consisting of an IPv4 host address, 16-bit process id, and 16-bit local value. In addition to the 16-bit process id not being standard across platforms (Mac OS >Leopard goes to 99999), the churn rate for the local value should be much higher than just 16 bits. This could be fixed to changing ORIGINATORID to a 32-bit process id and a separate value for the local value, which would be specified together in the DM event selector. I have to look into this more to see if this is a viable solution. This might need to go to three separate values to future proof it (i.e., ipv6). | 8 |
| 630 | DM-1354 | Install docker 1.1.2 in an ISL OpenStack CentOS instance, perform basic checks | Install docker 1.1.2 in an ISL OpenStack CentOS 6.5 instance, and perform basic checks such stopping and starting the docker daemon, changing default settings such as size limit of containers/images, pulling standard images from docker hub, starting containers from these images, etc. | 2 |
| 631 | DM-1355 | Re-arrange how Qserv directories are installed | we already touched question about installation directory structure at a meeting, maybe we can improve things by re-arranging how things are installed we are currently installing stuff into four directories: cfg, bin, lib, proxy to make it look more standard and to avoid clash with qserv source directories we could move cfg and proxy to a different location (something like share/qserv to make it more root-install-friendly in case we ever want to install under /usr) this change (if you want to do it) deserves separate ticket, do not do it in this ticket | 3 |
| 632 | DM-1358 | End-to-end demo fails to exit with the correct status when Warning would be correct. | The output of demo2012 results in an output file which is compared against a benchmarked file. Currently the comparison allows a deviation from the benchmark based "on the the number of digits in the significands used in multiple precision arithmetic"; that number is currently set to 11. An example using that setting is: @ Absolute error = 5.9973406400e-1, Relative error = 9.1865836000e-4 ##2566 #:25 <== 29.7751550737 ##2566 #:25 ==> 29.7478269835 Additionally, the current use returns: a 'Fatal' error if the code itself fails to execute correctly; a "Warning' error if the any of the benchmarked quantities do not meet the comparison criteria; 'Success' if the comparison meets all criteria. It is noted that the buildbot 'Warning' color indicator is not currently being displayed when the comparison fails. That is a coding error. This Issue will: * ensure that BUILDBOT_WARNING(S) causes the correct display color when appropriate. This Ticket has been split into two parts. This is part 1. Part 2 is DM-1379 | 2 |
| 633 | DM-1359 | improvements to PsfFlux | While using it as an example for the redesign of DM-829, I came up with some ideas for how to improve PsfFlux's handling of edge/bad pixels: - Add a flag indicating that at least one pixel was rejected - Add thresholds (number? fraction of PSF model size?) for how many non-rejected, non-edge pixels we need to even attempt a fit. | 1 |
| 634 | DM-1360 | Fix minor loose ends from new result plumbing | Some of the more minor issues raised in comments from DM-199 were left undone prior to merging. This ticket addresses them. For more information, please see: https://github.com/LSST/qserv/pull/2 . | 1 |
| 635 | DM-1362 | Edit pull interface and other Summer 2014 work into LSE-68 in Word | Deliverable: circulate a Word-based draft of LSE-68 in which the "push" interface is removed, and the "pull" interface is refined to include the guider and other Summer 2014 work. Note that the use of "pull" for the guider applies whether or not the proposed guider redesign is accepted. | 2 |
| 636 | DM-1363 | Avoid use of ~/.my.cnf (used by css watcher) | See https://jira.lsstcorp.org/browse/DM-1258?focusedCommentId=29230&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-29230. {{~/.my.cnf}} is used by css watcher, an optional tool used for monitoring css. It is a symlink to $QSERV_RUN_DIR/etc/my-client.cnf. The css watcher could use MySQL credentials located in ~/.lsst/qserv.conf (used by integration tests wich are a Qserv/MySQL client) | 2 |
| 637 | DM-1364 | replace "bad data" flag in SdssCentroid | SdssCentroid has a "bad data" flag that doesn't actually convey any information about what went wrong. This should be replaced with one or more flags that provide more information. | 3 |
| 638 | DM-1372 | Errors in testHealpixSkyMap.py | There is a failing unit test when healpy is supplied. The problem is that the method boundary is not defined in the version of healpy we supply for the sims, however boundaries does exist. If I replace boundary with boundaries, the test passes. | 1 |
| 639 | DM-1375 | Create Docker Image / Dockerfile for LSST Stack for CentOS6.5 | Make a Dockerfile for systematic generation of docker images using a Centos6.5 base image containing the LSST Stack (v9_2 at the moment) and library dependencies. | 2 |
| 640 | DM-1376 | Ensure that the partition package is C++11 clean and compiles on OSX 10.9 | The LSST buildbot infrastructure recently changed to building everything with --std=c++0x, which broke the partition package, and hence automated Qserv builds. While debugging this, I discovered that the partition package does not build on OSX 10.9, and considering how minimal its dependencies are, it really should. The OSX issue can be fixed by avoiding {{using boost::make_shared}}. The partition package should be cleaned up to avoid all use of {{using}}. If we decide to use C++11 in Qserv, then the codebase should also be modernized (in particular, there are use-cases for static_assert, nullptr, etc... ). | 2 |
| 641 | DM-1377 | Develop expanded and updated Information Security Program from NSF guidance/templates, using existing documents (e.g. LSE-99) as starting point | Develop expanded and updated Information Security Program from NSF guidance/templates, using existing documents (e.g. LSE-99) as starting point. Create master document and subordinate documents, allowing for existing AURA, site, and institutional polices to be incorporated as appropriate. JK: In PMCS this is Petravick D and Ephibian (LeClair L) | 56 |
| 642 | DM-1383 | Investigate deblending in one band followed by using the resulting templates in all bands | (See also [HSC-1025| https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1025]) One poor-man's approach to deblending multiple bands and visits is - Deblend in one band (or a combination, e.g. chi^2 band) using an SDSS-like algorithm that produces templates for each child - Take the templates (or possible model fits to those templates) and use them to deblend the objects in all other bands It will probably be necessary to include undetected objects in this fit (cf. [HSC-1023|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1023]), using point-source models. Note that this does not allow for the different seeing in each band. This is not obviously catastrophic for reasonably wide blends as the total flux in each pixel is conserved, and if only one template is important near an object's centre the exact form of the template is unimportant. | 40 |
| 643 | DM-1384 | Investigate deblending in one band followed by linear fits of models | (Cf. [HSC-1024|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1024]) One poor-man's approach to deblending multiple bands and visits is - Deblend in one band (or a combination, e.g. chi^2 band) - Fit models in that band - In all bands separately fit those models simultaneously to all object in the blend, allowing only a minimum of coefficients to float (ideally only amplitudes, but given the realities of real galaxies the bulge and disk will need to both be fit, or the Sloan Swindle components, or the Lensfit Swindle, or ...). The correct PSF for each image would be used. It will probably be necessary to include undetected objects in this fit (cf. [HSC-1023|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1023]), using point-source models. It is not clear how much of a poor-man's solution this actually is, or whether it will work quite well. | 40 |
| 644 | DM-1386 | Merge Footprints from different bands/epochs | The current concept of the deblender assumes that the inputs are - A merged set of Footprints that define which pixels are part of the blend - A merged set of Peaks within that merged Footprint Please generate these merged Footprints (which will be defined in (x, y) coordinates in the tract/patch coordinate system). | 5 |
| 645 | DM-1387 | Generate a master list of Objects given detections in multiple bands/epochs | Once we've detected Sources in multiple bands we need to merge the positions to generate Objects. This is a little complicated (or at least messy): - The positions have errors - If the seeing is different in different visits, objects may be blended in some but not all exposures - If we use more than one detection pass (e.g. smoothing when looking for faint objects, not smoothing for bright) this has similar-but-different consequences (but we should probably deal with in the per-band processing) - Objects move, so even if the positions are within the errors the motion may still be detectable | 5 |
| 646 | DM-1388 | Submit LCR for LSE-68 | Create an LCR, including a summary of changes, for LSE-68. | 2 |
| 647 | DM-1389 | Edit LSE-68 changes into EA | NULL | 2 |
| 648 | DM-1390 | Improve output from integration tests | Currently the integration tests print pages and pages of output, it is hard to see what failed and how. It'd be nice to clean that out, and instead, print some short summary, perhaps in a form of a summary table what queries has run, what succeeded, what failed etc. HTML format might be a reasonable way to display it. | 6 |
| 649 | DM-1394 | Eups 1.5.4 requires each new shell to source the eups setups.sh | eups v 1.5.4 requires each new shell to source ...eups/../bin/setups.sh. This requires the buildbot scripts: runManifestDemo.sh, create_xlinkdocs.sh, be updated to individually do that task. Add demo2012: bin/demo.sh . | 2 |
| 650 | DM-1395 | Reimplement CSS using JSON-packing, czar kazoo + c++ snapshotting phase 2 | The v1 implementation (DM-1249) transitions the zk access from the c++ layer into python, with code that understands how to unpack json-encoded data. This can coexist with the existing qserv_admin zk schema. v2 applies packing/unpacking logic in the creation/manipulation code in qserv_admin. v2 scope will be defined further, once DM-1249 enters review. Update: Phase 2 includes removal of c++ zk code and unification of Qserv css modules into a single place. | 10 |
| 651 | DM-1396 | Design CSS schema to support database deletion | Need to implement deleting databases. Deliverable: a design of the system that will be capable of deleting a distributed database including all copies of that database on all workers, all replicas of all chunks are deleted. It should be possible to "create database x" at any time later. | 2 |
| 652 | DM-1400 | Improve documentation of pixel systems in obs_lsstSim | There is not much documentation of the coordinate systems in use by CameraGeom. This is by nature documentation that is instrument specific, so should go in the obs_ package for each instrument. | 1 |
| 653 | DM-1404 | Create suite of Dockerfiles / docker images for LSST Stack for ubuntu, CentOS | Building on issues DM-1317 and DM-1375 where initial images and Dockerfile's were constructed, we can now use these Dockerfile's as prototypes to extend the set of Dockerfiles & images. We observe that by making simple (scriptable) edits to the initial Dockerfile, we can run 'docker build' to make docker images for several combinations of OS and base compiler gcc version. | 2 |
| 654 | DM-1405 | Prepare a SL6x openstack image for with Qserv development environment | Qserv packaging procedure requires to often rebuild Qserv and relaunch integration tests. IN2P3 Openstack platform offer next virtual machines : {code:bash} [fjammes@ccage030 ~]$ nova flavor-list {code} | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | | 1 | m1.tiny | 512 | 0 | 0 | | 1 | 1.0 | True | | 15 | cc.windows.small | 4096 | 20 | 0 | | 2 | 1.0 | True | | 16 | cc.windows.xlarge | 8192 | 50 | 0 | | 4 | 1.0 | True | | 2 | m1.small | 2048 | 10 | 20 | | 1 | 1.0 | True | | 3 | m1.medium | 4096 | 10 | 40 | | 2 | 1.0 | True | | 4 | m1.large | 8192 | 10 | 80 | | 4 | 1.0 | True | | 5 | m1.xlarge | 16384 | 10 | 160 | | 8 | 1.0 | True | | 6 | cc.lsst.medium | 4096 | 20 | 40 | | 2 | 1.0 | False | | 7 | cc.lsst.large | 16384 | 20 | 160 | | 8 | 1.0 | False | | 9 | cc.lsst.xlarge | 40000 | 20 | 160 | | 20 | 1.0 | False | cc.lsst.xlarge would allow a quick build/test of new Qserv release. | 5 |
| 655 | DM-1407 | Add return code for integration tests | Integration test have to returns non-zero when failing. This will ease use of CI or debugging tools ({{git bisect}}, buildbot). | 1 |
| 656 | DM-1411 | Stand up netem server with Dell PE 1950 | NULL | 4 |
| 657 | DM-1424 | Create persistent volume of Cinder block storage and attach to instance | We create a persistent volume of Cinder block storage and attach to working instance. When it was created, the instance does have a specified amount of ephemeral disk, but this disk will be destroyed with the instance. We want to test that we can create a persistent volume of block storage, attach it to an instance, format the storage with a file system, and mount the volume for use with processing, where data/output can be retained after the instance is destroyed. | 1 |
| 658 | DM-1426 | Document how to switch Qserv or dependency in eups | DM-1338 may allow to switch Qserv or dependency version using eups, without having to re-configure Qserv (i.e. without any change in QSERV_RUN_DIR). There may have conditions for this to work. That's why it's reasonable to get some user feedback on DM-1338 before documenting this feature. | 2 |
| 659 | DM-1428 | Add return codes for qserv-check-integration.py and qserv-testunit.py | NULL | 1 |
| 660 | DM-1429 | Improve Qserv Configuration Procedure | Based on input from review of DM-1338, and discussions at hangout https://confluence.lsstcorp.org/display/DM/LSST+Database+Hangout+2014-10-29: implement -keepdata option. * --keepdata will preserve qserv_meta.conf and my.cnf. Regarding the latter, this is necessary because user might have customized data_dir. * The structure of qserv_meta.conf and my.cnf must remain the same between existing version of Qserv and the to-be-configured version of Qserv. It is users's responsibility to ensure the structure did not change. Note in particular, the template files (with the exception of my.cnf) should not be customized | 7 |
| 661 | DM-1431 | Process sample sdss data with LSST Stack in a Docker container in OpenStack | Process sample sdss data, starting with the lsst_dm_stack_demo, with LSST Stack in a Docker container in an OpenStack instance. | 2 |
| 662 | DM-1432 | SUI define local hardware needs to host the test DB and application servers | We need have Qserv and the test DB in IPAC to do development and test locally, to access the data through Qserv APIs. | 10 |
| 663 | DM-1433 | SUI install Qserv and test DB, enable access to catalogs for SUI team members | Install Qserv and test DB in local hosts. Enable SUI team members to access the catalogs. | 6 |
| 664 | DM-1434 | State diagram for jobs in progress | Build a state diagram showing job progress throughout a run. | 13 |
| 665 | DM-1439 | afw tests use the same name for a dummy file: test.fits | The afw package uses a file named: test.fits in multiple testers. If the user sets up the build to use multiple CPU (-j #), then there is the risk that the shared filename will be affected by more than one tester at a time. In the case which provoked this Issue, the tester: testSimpleTable.py, reported that the file was missing. A simple rerun managed to get past the error. I recommend that the different testers use uniquely named demo files. | 1 |
| 666 | DM-1440 | Github Transition: Naming conventions for repositories | The Simulations team has requested that repos in general and the numerous DM repos in particular are prefixed in a way that would make fitering them out of searches and display easy (for example, their repos are prefixed sims_*) This would be an evident useability aid to DM developers and outside contributors too. Obtain a decision on how to allow users to quickly isolate repositories they are interested in. | 3 |
| 667 | DM-1441 | Github Transition: Storage of large data files | Github currently has fixed 100MB per blob or 1GB per repo limits. Sims has at least one file in its repo whose history exceeds this (sims_meas) and this has been raised as an issue before. Obtain a decision on a suitable way forward for the time being that would allow the upload of the repositories on Github. Experienced to be reviewed in a few months time to consider whether has proved satsifactory. | 4 |
| 668 | DM-1442 | Github Transition: Stash-stored pull requests, extraction | Extract comments from Atlassian Stash made during pull requests /code reviews and their associated SHA1s if it all possible. | 4 |
| 669 | DM-1443 | Github Transition: pull request discussion, retention - proof of concept | Store review comments / PR discussion into relevant git repositories Code & process to do this on a continuous basis post-transition will be a different issue, | 1 |
| 670 | DM-1444 | Test absence of individual components | Test absence of individual components of the AP simulator. Bring each down, run the system, and restart just those components to see if the system still operates as expected | 2 |
| 671 | DM-1445 | check if Qserv compiles with C++11 | NULL | 1 |
| 672 | DM-1446 | Fine tune czar and worker database initialization | For now, Qserv databases are the same on the master and on the worker. It means that czar tables are created on the worker and vice-versa. This ticket aims at creating the right tables at the right places, and also at sharpening permissions on these tables. This is done by splitting configuration script qserv-czar.sh, which configure a mono-node instance, in two scripts: - qserv-czar.sh, for configuring the czar - qserv-worker.sh, for configuring workers. see https://confluence.lsstcorp.org/display/DM/LSST+Database+Hangout+2014-10-29 Please note that this code can't be validated with mono-node integration tests (whereas it doesn't break them): indeed it requires a mono-node installation with 2 MySQL instance, one for the worker and one for the master. Updating the mono-node configuration and integration tests with such a feature would make them far more complex. A quick and dirty, hard-coded, testbed is available in u/fjammes/DM-1446-test. It has been used successfully to test this ticket. This ticket will also be validated during next Qserv install on in2p3 clusters. | 4 |
| 673 | DM-1447 | Improve spatial-selection flexibility by parsing ptInSphBox-like syntax instead of qserv_areaspec_box | Note: it is not clear that we should do this. This is an idea to change the syntax for spatial area selection. Currently, we have SELECT...FROM...WHERE qserv_areaspec_box(...) ... This forces the area selection to apply the box (or appropriate) cut on the tables referenced in the FROM list that are partitioned, with those columns. It would be nice to allow spatial selection based on tables (and columns?) explicitly specified by the user, and then compute the appropriate chunk coverage and WHERE clause conditions and predicates. This is a pretty big deal, and would affect all spatial-select user queries, though the compatibility can be mitigated by supporting both syntaxes. | 100 |
| 674 | DM-1448 | Move code for mock images into afw so it reusable. | There is some code in the exampleUtils in $IP_ISR_DIR/examples that could be of wider use. Specifically there is code to generate mock darks, flats, and raw data from a mock camera. There is also code to generate a mock dataRef. It could be used more widely if moved someplace else. Russell suggested afw.cameraGeom.utils. | 1 |
| 675 | DM-1449 | newinstall.sh should check that python2 is available | The standards now suggest using: {code} #!/usr/bin/env python2 {code} instead of {code} #!/usr/bin/env python {code} This doesn't work with (at least) Mac OSX 10.9 system python. newinstall.sh should check if this works and suggest a fix (creating a symlink). | 1 |
| 676 | DM-1452 | Detect lua version in admin/templates/configuration/etc/init.d/mysql-proxy | NULL | 1 |
| 677 | DM-1453 | Add data versions to Zookeeper | Track versions of data inside zookeeper, and detect from Qserv code if Qserv code is compatible with given format of data. | 2 |
| 678 | DM-1454 | LOE - Week ending 10/31/14 | NULL | 8 |
| 679 | DM-1455 | LOE - Week ending 10/24/14 | NULL | 6 |
| 680 | DM-1456 | finish porting meas_algorithms unit tests | NULL | 4 |
| 681 | DM-1461 | C++ Redesign -- Result definition for custom algorithms | Additions to Jim's redesign to make it easier to define custom results. | 3 |
| 682 | DM-1462 | Add NaN check to PixelFlags | The test of PixelFlags in measureSources.py (from measAlg) requires a check to be sure that the center inputs are not NaN. | 1 |
| 683 | DM-1463 | SdssShape shiftMax config item is being ignored | The code we ported from meas_algorithms sets the maxShift to 2 without regard to the config item which is supposed to set that value. | 1 |
| 684 | DM-1464 | Design Review prep for C++ redesign | Write up the design developed on DM-829 and push it through review. | 1 |
| 685 | DM-1469 | Github Transition Plan: Reverse mirror for beta-tester repositories | Test the reverse mirror for beta-testers. Straw man: 1. Break the mirror for anybody beta-testing github workflow 2. Mirror back to new gitolite area: "mirror") Method: https://help.github.com/articles/duplicating-a-repository/ | 1 |
| 686 | DM-1471 | Rewrite obs_test | I want to use obs_test to test processCcd, but it contains no imaging data. I will replace the current data with new imaging data (a portion of one LsstSim CCD), add a camera and rewrite the mapper (subclassing CameraMapper). This also requires rewriting the unit tests in obs_test and in one other package that uses obs_test. | 10 |
| 687 | DM-1473 | Execute multi-platform lsst_dm_stack_demo test with Fig orchestration | In DM-1431 we demonstrated processing of sample data within Docker containers within an OpenStack instance. Data was processed for containers based on CentOS6.5, Ubuntu 13.10, Ubuntu 14.10. We consider a multi-platform "testing" scenario, where we process the sample data in numerous containers based on various platforms/OSs all simultaneously on an OpenStack instance. This scenario entails starting up and managing multiple Docker containers. Fig (http://www.fig.sh) is a new tool for starting up and managing services via Docker containers. It is a common use case with Docker to have individual components of an application each run separately in a container (resulting in numerous containers running on a node or cloud env, i.e., a 'Docker stack'), with the containers having linkages/dependencies to be managed. Fig may be used to encode the relationship between the containers in the Docker stack, and to start up such a set of containers. We install and examine Fig, and apply Fig to our multi-platform "testing" scenario. | 4 |
| 688 | DM-1475 | Fix 2014_09 documentation | Replace {{NEWINSTALL_URL=http://sw.lsstcorp.org/pkgs/}} with: {{NEWINSTALL_URL=http://sw.lsstcorp.org/eupspkg/}} and {{eups distrib install qserv_distrib 2014_10.0}} with: {{eups distrib install qserv_distrib -t 2014_10}} | 1 |
| 689 | DM-1476 | Secure MySQL root password in configuration templates | MySQL password in written in multiple file during configuration procedure. One single file (QSERV_RUN_DIR/tmp/my.cnf) should be used, and removed at the end of configuration procedure. qserv-meta.conf also contains MySQL password and should be also secured (move password to qserv-configure.py cmd line?). | 4 |
| 690 | DM-1477 | Study fig to manage Qserv cluster | http://www.fig.sh/ coordinate with @GregDaues who is also investigating fig. | 5 |
| 691 | DM-1480 | Buildbot master takes exception when exiting from mail notifier after dynamic email sent. | Buildbot master exits without posting the required statically-addressed email notification if a dynamically-addressed was sent. This fix needs to ensure that the required (by buildbot specification) static email is sent even if it has to be directed to a dead-letter box (which it is). | 3 |
| 692 | DM-1481 | Discover/learn what others are doing in astronomy software | Attend the annual ADASS conference to keep up with the software development in the astronomy community. Trey Roby, Tatiana Goldina, Xiuqin Wu plan to attend the ADASS 24. | 20 |
| 693 | DM-1482 | SUI: study other plot packages | Firefly uses clientsidegchart at https://code.google.com/p/clientsidegchart/ for XY plot. This package has not been updated since 2010. We need to find out more about other plotting packages and consider if we need to switch. The candidates are: 1. flot plotting library for JQuery. http://www.flotcharts.org/ 2. Highcharts JS at http://www.highcharts.com/ | 10 |
| 694 | DM-1488 | Finalize the Design of Query Metadata | Fine-tune the design outlined in DM-1251 (metadata for capturing information about running queries in Qserv.). | 6 |
| 695 | DM-1489 | Modify czar to interact with query metadata | Modify czar code: it should interact with the query metadata (store information for long running queries, retrieving). | 6 |
| 696 | DM-1490 | FY18 Design and Implement Query Cost Estimate | Design and implement system that will estimate query cost. In particular, we will need to know if the query is interactive, or should be scheduled on shared scan, which shared scan etc. Estimating cost will likely involve looking at number of chunks involved, number of joins, and complexity of math operations. Consider extending API and allow users to specify a hint. | 79 |
| 697 | DM-1493 | Add queries on partitioned table for Qserv integration test dataset #3 | Qserv test dataset #3 runs only queries on non-partitioned tables. Adding queries on partitioned table would increase drastically the coverage of these tests. Please note that a long-term solution could be to launch all the *.FIXME queries and to have a more detailed report. For example define queries which must pass for the integration test to succeed, and test queries which may pass and log their results in a report. | 4 |
| 698 | DM-1495 | Allow newinstall.sh to run in batch mode without installing Anaconda | Installing Qserv on a cluster without internet access requires newinstall.sh to run in batch mode without installing Anaconda (whose install requires internet access). So Anaconda should have a {{-anaconda=yes|no}}. Usefulness of git {{-git=yes|no}} option, against full batch option answering "yes" everywhere could also be studied. | 2 |
| 699 | DM-1497 | Package Qserv mono-node instance in Docker | Learn Docker basics and then package a Qserv mono-node instance. | 5 |
| 700 | DM-1498 | Package Qserv master and worker instance in Docker | Learn Docker basics and then package a Qserv mono-node instance. | 5 |
| 701 | DM-1505 | confusing error message when enabling unregistered items in RegistryField | pex_config seems to split out this confusing error message when trying to enable (i.e. append to .names) a registry item that doesn't exist: {code:hide-linenum} File "/home/lam3/tigress/LSST/obs_subaru/config/processCcd.py", line 51, in <module> root.measurement.algorithms.names |= ["jacobian", "focalplane"] File "/tigress/HSC/LSST/lsstsw/anaconda/lib/python2.7/_abcoll.py", line 330, in __ior__ self.add(value) File "/tigress/HSC/LSST/lsstsw/stack/Linux64/pex_config/9.0+26/python/lsst/pex/config/configChoiceField.py", line 72, in add r = self.__getitem__(value, at=at) AttributeError: 'SelectionSet' object has no attribute '__getitem__' {code} | 1 |
| 702 | DM-1506 | Support new version of newinstall.sh | newinstall.sh now creates loadLSST.bash instead of loadLSST.sh. This has to be taken in account in Qserv automated install script: qserv-install.sh and in Qserv documentation. | 1 |
| 703 | DM-1508 | Reverse image slicing doesn't work as expected | The lsst.afw.image image-like classes support slicing, but reverse slicing does not work as expected. Here are some examples: {code} from lsst.afw.image import ImageF im = ImageF(10, 10) im[:,:].getDimensions() # is (10,10) as expected im[0:10, 0:10].getDimensions() # is (10,10) as expected im[::-1, ::-1] # should be the data with x and y reversed, but fails with: Box2I(Point2I(-2,-2),Extent2I(12,12)) doesn't fit in image 10x10 im[9:1:-1, :].getBBox() # starts at 0,0, as expected, but is 10x10 instead of 9x10 {code} | 1 |
| 704 | DM-1509 | Remove unnecessary config/install steps | Some existing installation configuration steps are not needed any more. For instance, we don't need $RUN/q or $RUN/result any more-- these paths are handled directly by qserv code, so the filesystem lookup shouldn't ever take place. Completion of this ticket should remove unnecessary steps and concepts from installation, simplifying future maintenance. | 2 |
| 705 | DM-1510 | C++ Standards Rule 3-9 needs rewrite | Jim Bosch reviewed Section 3 of the C++ Standard. He noted this change which I felt warranted SAT clarification: 3-9: This is a confusing conflation of two entirely different concepts: it seems to discourage all global variables, regardless of whether they're in a namespace, but only to discourage free functions when they aren't in namespace. If that reading is correct, I think both are sensible recommendations, but they need to be clarified, and probably split up. If the reading should be that all free functions are discouraged, that'd be a terrible rule we violate all over the place. If the reading should be that global variables are only discouraged when not in a namespace, that needs to be clarified (and IMO it could be bumped up to a complete prohibition). Refer to: https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908685 Rule 3.9. The current Rule states: 3-9. Global variables and functions SHOULD be avoided and if used MUST always be referred to using the '::' operator. _________________________________________ ::mainWindow.open(), ::applicationContext.getName(), ::erf(1.0) __________________________________________ In general, the use of global variables should be avoided. Consider using singleton objects instead. Only use where required (i.e. reusing a framework that requires it.) See Rule 5-7 ( .https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908706#C++Statements-5-7 ). Global functions in the root namespace that are defined by standard libraries can often be avoided by using the C++ versions of the include files (e.g. "#include <cmath>" instead of "#include <math.h>"). Since the C++ include files place functions in the std namespace, "using namespace std;", which is permitted by Rule 5-41, will allow these functions to be called without using the '::' operator. In cases where functions are only available in the C include files, the '::' operator must be used to call them. This requirement is intended to highlight that these functions are in the root namespace and are different from class methods or other namespaced free functions. | 1 |
| 706 | DM-1514 | calling extend with a SchemaMapper should support positional arguments | Calling {{catalog.extend(other, mapper)}} isn't equivalent to {{catalog.extend(other, mapper=mapper)}} because the second argument is the boolean {{deep}}. When a SchemaMapper is passed as the second argument, we should recognize it for what it is. | 1 |
| 707 | DM-1515 | sconsUtils fails to identify Ubuntu's gcc | On Ubuntu 12.04, gcc --version says: {code:hide-linenum} gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3 Copyright (C) 2011 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. {code} This apparently isn't quite what sconsUtils expected, because it says: {code:hide-linenum} scons: Reading SConscript files ... Checking who built the CC compiler...(cached) error: no result CC is unknown version unknown {code} Happily, everything seems to work anyway, as the fall-back options for the unknown compiler work fine with this one. | 1 |
| 708 | DM-1516 | Use eups swig instead of system swig during Qserv build | Qserv build system use system swig, it should be fixed to use eups swig. | 3 |
| 709 | DM-1518 | Add support for "SET @@session.autocommit" | The SUI team is using JDBC connector and queries like "SET @@session.autocommit = {0}".format(switch) are derailing everything. We should add support for these queries. | 2 |
| 710 | DM-1519 | check out FIrefly package, build and run an applicaiton | Get familiar with FIrefly package, build and run an application. Understand the coordinate grid overlay on an image. | 10 |
| 711 | DM-1520 | Fix confusing error message | Selecting from a table that does not exist, e.g. something like: select count(*) from whdfd produces a strange error: ERROR 4110 (Proxy): Qserv error: 'Unknown error setting QuerySession' | 1 |
| 712 | DM-1521 | Fix "stripes and substripes must be natural numbers" bug | on user side: {code} ERROR 4110 (Proxy) at line 1: Qserv error: Unexpected error: (<class 'lsst.qserv.czar.config.ConfigError'>, ConfigError(), <traceback object at 0x26b8b90>) {code} in czar log: {code} 1114 22:30:10.155 [0x7fafcefab700] ERROR root (app.py:370) - Unexpected error: (<class 'lsst.qserv.czar.config.ConfigError'>, ConfigError(), <traceback object at 0x26b8b90>) Traceback (most recent call last): File "/usr/local/home/becla/qservDev/Linux64/qserv/2014_09.0/lib/python/lsst/qserv/czar/app.py", line 360, in __init__ self._prepareForExec() File "/usr/local/home/becla/qservDev/Linux64/qserv/2014_09.0/lib/python/lsst/qserv/czar/app.py", line 437, in _prepareForExec self._addChunks() File "/usr/local/home/becla/qservDev/Linux64/qserv/2014_09.0/lib/python/lsst/qserv/czar/app.py", line 535, in _addChunks self._computeConstraintsAsHints() File "/usr/local/home/becla/qservDev/Linux64/qserv/2014_09.0/lib/python/lsst/qserv/czar/app.py", line 529, in _computeConstraintsAsHints self.pmap = self._makePmap(self.dominantDb, self.dbStriping) File "/usr/local/home/becla/qservDev/Linux64/qserv/2014_09.0/lib/python/lsst/qserv/czar/app.py", line 478, in _makePmap raise lsst.qserv.czar.config.ConfigError(msg) ConfigError: "Partitioner's stripes and substripes must be natural numbers." {code} To reproduce: {code} qserv-check-integration.py --case-no 01 --load {code} then {code} mysql --port 4040 LSST -e "select count(*) from Object" {code} | 2 |
| 713 | DM-1524 | Switch readMetadata' return value from PropertySet to PropertyList | It'd be useful if readMetadata would return PropertyList instead of PropertySet, because in many situations order matters. For details, see discussion in DM-1517 | 2 |
| 714 | DM-1525 | Investigate halos around stars | Andrew Becker reports on dm-users: {quote} We have been using the LSST stack to reduce CFHT data at UW, and have come across a potential bug in the DM software. I don't see how this could be something intrinsic to the data, instead it seems like it could be a bug in the software triggered by its application to new data. I have a how to reproduce at NCSA at /lsst/home/becker/CFHT. Follow commands in the SETUP file to get an up-to-date version of the stack (b449) and a modified version of ip_isr (a hack to avoid overscan correction) and a private branch of obs_cfht (u/krughoff/ossos_support). If you run the commands in SOURCE_ME it will run the processCcd on a single CCD, in 2 modes, yielding 2 output directories. The second call merely sets the bin sizes to be the same amongst all the various background subtractions, but is sufficient to trigger the bug. A basic script included in the directory differences the 2 output calexps (a straight -=) and ds9 is used to view the diff. If you pan to e.g. 1845,546 you'll see a feature like attached in the difference (out1 on the left, out2 in the middle, diff on the right). This can be seen in various other parts of the image, as irregular annuli around bright blended objects. It suggests to me that during measurement some substamp manipulations are leaking into the original image. But you can kind of see in the center image that the second run seems to trigger the issue. {quote} And then: {quote} So to summarize: export LSSTSW=~lsstsw source ~lsstsw/bin/setup.sh setup lsst_apps -t b449 setup -k -r /nfs/lsst/home/becker/LSST/DMS/obs_cfht setup -k -r /nfs/lsst/home/becker/LSST/DMS/ip_isr setup -k -r /nfs/lsst/home/becker/LSST/DMS/processFile cd /nfs/lsst/home/becker/CFHT will set up your environment. The following is sufficient to recreate the bug, and create postIsrCCD images for processFile: processCcd.py input/ --id visit=1612606 ccd=8 --output out1/ --config isr.doAssembleDetrends=False isr.fringeAfterFlat=False isr.fringe.pedestal=False isr.doBias=False isr.doFlat=False isr.doFringe=False isr.doWrite=True calibrate.doPhotoCal=False calibrate.doAstrometry=False doDeblend=True processCcd.py input/ --id visit=1612606 ccd=8 --output out2/ --config isr.doAssembleDetrends=False isr.fringeAfterFlat=False isr.fringe.pedestal=False isr.doBias=False isr.doFlat=False isr.doFringe=False isr.doWrite=True calibrate.doPhotoCal=False calibrate.doAstrometry=False doDeblend=True detection.background.binSize=512 calibrate.detection.background.binSize=512 calibrate.background.binSize=512 Following are the calls I made to processFile.py (using my modified version of the processFile.git package), and which return images whose difference is exactly 0. So I don't really get whats happening with processFile, but it doesn't seem to be accepting my command-line config changes. processFile.py out1/postISRCCD/13AP06/E+0+0/2013-03-11/r/postISRCCD-1612606-08.fits --outputCalexp calexp1.fits -c calibrate.doPhotoCal=False calibrate.doAstrometry=False doDeblend=True calibrate.repair.doCosmicRay=False processFile.py out2/postISRCCD/13AP06/E+0+0/2013-03-11/r/postISRCCD-1612606-08.fits --outputCalexp calexp2.fits -c calibrate.doPhotoCal=False calibrate.doAstrometry=False doDeblend=True calibrate.repair.doCosmicRay=False detection.background.binSize=512 calibrate.detection.background.binSize=512 calibrate.background.binSize=512 processFile.py out2/postISRCCD/13AP06/E+0+0/2013-03-11/r/postISRCCD-1612606-08.fits --outputCalexp calexp3.fits -c calibrate.doPhotoCal=False calibrate.doAstrometry=False doDeblend=True calibrate.repair.doCosmicRay=False detection.background.binSize=1024 calibrate.detection.background.binSize=1024 calibrate.background.binSize=1024 {quote} | 2 |
| 715 | DM-1526 | Update processFile to use new measurement framework | processFile.py uses the old measurement framework in meas_algorithms, but all the other components expect it to be using the new measurement framework in meas_base. | 4 |
| 716 | DM-1527 | Draft security risks into the Center's template | The Cyber security center has provided a risk template consistent with their templates, The Center attempted to populate their templates with material from LSE 99 to the templates, The impedance mis match was too large, The way forward is seem as attempting a high level decomposition of risk into the templates. | 4 |
| 717 | DM-1528 | Identify potential KVM hardware | Identify potential KVM hardware that would meet our needs. e.g. a current version of the Avocent or Dell KVMs used at NCSA. | 1 |
| 718 | DM-1529 | Reorganize docker image repositories and align with github | A heterogeneous collection of docker images have been accumulating within the public docker repository daues/lsstdistrib . Such a heterogeneous collection prevents the assignment of a "latest" tag to allow users to easily obtain the most recent image for a particular item (detailed version numbers, labels currently required.) Thus we should break out the single repository into multiple repositories where are "latest" tag will be effective. We also make github repositories of matching names to hold the Dockerfiles which produced images (a common pattern for github/dockerhub usage, especially with automated builds; so we start this practice.) | 2 |
| 719 | DM-1530 | Test if IPMI can replace KVM | Can we use IPMI in place of a KVM? Can we reliably do the following across our various server vendors? - power cycle hung systems - view and use text console - view and use remote gui console - boot from a remote ISO | 3 |
| 720 | DM-1531 | Create a LSST base CentOS image | NULL | 2 |
| 721 | DM-1532 | Document use of LSST image for CentOS | NULL | 1 |
| 722 | DM-1533 | Gather Open Stack needs/requirements from DM team | NULL | 9 |
| 723 | DM-1534 | Assist DM team in accessing test open stacks | NULL | 1 |
| 724 | DM-1538 | Fix qserv_testdata documentation | qserv_testdata relies on sconsUtils, and its build procedure has to be clearly documented. | 1 |
| 725 | DM-1539 | Add support for mysql JDBC driver | SUI which rely on JDBC fail because they internally issue some queries that are not yet supported by Qserv. Need to patch it (in the short term), and add proper support (in the long term). This story covers the patching only. The queries that upset Qserv are listed below. {code} SHOW VARIABLES WHERE Variable_name ='language' OR Variable_name = 'net_write_timeout' OR Variable_name = 'interactive_timeout' OR Variable_name = 'wait_timeout' OR Variable_name = 'character_set_client' OR Variable_name = 'character_set_connection' OR Variable_name = 'character_set' OR Variable_name = 'character_set_server' OR Variable_name = 'tx_isolation' OR Variable_name = 'transaction_isolation' OR Variable_name = 'character_set_results' OR Variable_name = 'timezone' OR Variable_name = 'time_zone' OR Variable_name = 'system_time_zone' OR Variable_name = 'lower_case_table_names' OR Variable_name = 'max_allowed_packet' OR Variable_name = 'net_buffer_length' OR Variable_name = 'sql_mode' OR Variable_name = 'query_cache_type' OR Variable_name = 'query_cache_size' OR Variable_name = 'license' OR Variable_name = 'init_connect' SELECT @@session.auto_increment_increment SET NAMES latin1 SET character_set_results = NULL SET autocommit=1 SET sql_mode='STRICT_TRANS_TABLES' {code} | 2 |
| 726 | DM-1540 | Add support for BIT columns | The mysql/SchemFactory code has incomplete logic for dealing with columns that have BIT type - in particular, it doesn't properly handle the length of a bit field, and it sets the SQL type to "BIT?", which causes table creation failures later on. Fixing this requires modifying SchemaFactory, and making sure that bit values are transmitted properly. | 4 |
| 727 | DM-1541 | Add support for transmitting [VAR]BINARY column data | The code that pulls data out of a {{MYSQL_ROW}} and puts it into a protobuf {{RowBundle}} does not handle binary data correctly. See [_fillRows|https://github.com/LSST/qserv/blob/master/core/modules/wdb/QueryAction.cc#L217] for the relevant code. The issue is that the generated {{add_column(const char*)}} member function of {{RowBundle}} treats the input as C-style NULL-terminated strings. But in the case of {{BINARY}} column data (and {{VARBINARY}}/{{BLOB}} variants, maybe also {{BIT\(n\)}}), the contents can contain embedded NULLs. We are currently using such columns for user defined types in MySQL (e.g. image bounding polygons), so it's important to get this right. On the protobuf side the fix is as simple as calling {{add_column(const char* value, int size)}} instead. I'm not a MySQL C API expert, but the size will presumably have to be obtained/derived from the corresponding {{MYSQL_FIELD}}. | 8 |
| 728 | DM-1542 | Drawing speed will detect and optimize for large datasets | NULL | 10 |
| 729 | DM-1543 | Finished Background conversion | NULL | 2 |
| 730 | DM-1544 | Set up GIT hub for ipac firefly | NULL | 18 |
| 731 | DM-1545 | Span-based shrink operations for Footprint | Analogous to DM-1128, but shrinking rather than growing footprints. | 5 |
| 732 | DM-1547 | Install docker in an ISL OpenStack Ubuntu instance, perform basic checks | NULL | 1 |
| 733 | DM-1551 | Prototype HTM-based spatial binning to visualize large number of catalog sources | Currently we are using a generic 2-d binning algorithm, that is finding minimum and maximum values of the two columns to be visualized and bins the values into 2-d grid with the specified number of grid cells. This algorithm distorts data in the pole regions and whenever data are on both sides of ra=0. More smart binning based on a spatial index is necessary when reducing the number of (ra,dec) entries intended for visualization. | 16 |
| 734 | DM-1552 | Resolving QServ database configuration/connectivity issues | To start the development I should be able to connect to the QSERV development database via VPN and run simple queries. Ideally, I'd like to be able to connect to QSERV with JDBC, view the data, run spatial queries. I also need an access to data dictionary to interpret data. | 10 |
| 735 | DM-1553 | Evaluate SDSS catalog access | SDSS allows two ways to access their catalog data: via HTTP Post service, accessible to anonymous users, and via CasJobs services, which require having an account. Evaluate how SDSS catalog data can be accessed from an application by prototyping single and multiple target searches accessing SDSS services. | 10 |
| 736 | DM-1556 | FY17 Add Support for User Upload Tables for Qserv | Users will need to be able to upload a list of "things" to workspace, then request "repeat a given query for each "thing" from the uploaded list". Example of "things": ra/dec points, ra/dec points + distances, object ids, object names, bounding boxes, etc. In IPAC terminology, this is called "table upload queries". Note, we will need to assign some sort of unique id to such list. Results must contain information which result correspond to which "thing". E.g, if user asks for neighbors near (1,1), (4,4), it is not enough to just return list of neighbors, we need to tell which neighbor is for which point. In Qserv, that means that user will upload a special table with these "things", and then issue a query that involves join against that table. We need to evaluate and agree on how generic these upload tables can be. We might need to do some special optimizations (e.g., related to spatial searches). | 53 |
| 737 | DM-1557 | FY17 User Upload Tables for ImgServ | Users will need to be able to upload a list of "things" to workspace, then request "repeat a given query for each "thing" from the uploaded list". Example of "things": ra/dec points, ra/dec points + distances, object ids, object names, bounding boxes, etc. In IPAC terminology, this is called "table upload queries". Note, we will need to assign some sort of unique id to such list. Results must contain information which result correspond to which "thing". E.g, if user asks for neighbors near (1,1), (4,4), it is not enough to just return list of neighbors, we need to tell which neighbor is for which point. In Qserv, that means that user will upload a special table with these "things", and then issue a query that involves join against that table. We need to evaluate and agree on how generic these upload tables can be. We need to decide on format / location: csv? Table in database? We might need to do some special optimizations (e.g., related to spatial searches). | 53 |
| 738 | DM-1560 | A better coordinate grid overlay | Some of the grid lines are not drawn right and the labels are not in the locations. | 10 |
| 739 | DM-1561 | Research Javascript Frameworks: General Overview | Begin looking into JS frameworks. Start to look into AngularJS, try to write some sample code. Attempt to understand the main concepts. Gat an overview of the others out there. | 12 |
| 740 | DM-1562 | Work with the database group to define initial concepts for backend interface API | We are starting to gel around some ideas. | 10 |
| 741 | DM-1563 | Improve management of integration test datasets description | Currently data set description (i.e. data file extension, compressed extension, schema file extension) is hard-coded in python/lsst/qserv/tests/datareader.py This could be improve (standard format for test dataset, meta service, meta-configuration file, ...) | 5 |
| 742 | DM-1564 | Implement drawing only active tab with new data model | NULL | 6 |
| 743 | DM-1566 | LOE - Week ending 11/07/14 | NULL | 7 |
| 744 | DM-1567 | LOE - Week ending 11/14/14 | NULL | 6 |
| 745 | DM-1568 | LOE - Week ending 11/21/14 | NULL | 8 |
| 746 | DM-1569 | LOE - Week ending 11/28/14 | NULL | 12 |
| 747 | DM-1570 | Create integration test case using data duplicator | Integration tests should provide a new test case which use sph-duplicate in partition package. | 6 |
| 748 | DM-1571 | Setup Qserv for SUI tests | Setup Qserv on lsst-db2 with and load some reasonable data set (perhaps PT 1.2). One potential caveat: we need to setup access for some accounts that are ideally other than our internal qsmaster. | 2 |
| 749 | DM-1572 | Test deblended CModel colors | Using HSC data, examine color-color and color-magnitude diagrams with deblended CModel magnitudes. Investigate outliers by looking at images and deblended HeavyFootprints. | 8 |
| 750 | DM-1573 | Basic validation of LSST pipeline on HSC data | Get the pipeline running on HSC data to the point where nothing is obviously wrong. | 8 |
| 751 | DM-1574 | add support for "freeze-drying" measurement failures | We should make it easy for users to inspect and capture problems in measurement algorithms into an on-disk format that allows them to be reproduced later with minimal setup (ideally, the package would require no access to the original data or configuration). While this issue is mostly concerned with capturing problems in measurement algorithms, an extension to capture deblender failures should be considered as a future extension. | 6 |
| 752 | DM-1575 | Support Mac OS in scisql-deploy.sh | cf. Andy Connolly message: {quote} Just as an FYI on my mac scisql-deploy.py was looking for 0.3.4/lib/libscisql-scisql_0.3.so but there is only 0.3.4/lib/libscisql-scisql_0.3.dylib {quote} | 1 |
| 753 | DM-1576 | Sanitize AstrometryTask interface | Currently the AstrometryTask and Astrometry class share work. E.g. distortion is done in AstrometryTask but matching is done in Astrometry. AstrometryTask also makes assumptions about what fields are available in the solver config. The AstrometryTask interface should be sanitized so that it can be used as a thin wrapper for calling any astrometry solver. Top level config params should go in the AstrometryTaskConfig and solver level work should be done in the solver class and configured at the solver level. | 10 |
| 754 | DM-1577 | Rework Astrometry class | The Astrometry class shares information upstream with AstrometryTask. This should be factored out so that the Astrometry class can be configured and called via a single well known method (solve?). One thing the Astrometry class that is needed by down stream processing (PhotoCal) is to match sources. This is currently a private method, but should be made public so that it can be used without running AstrometryTask. | 4 |
| 755 | DM-1578 | Move photocal out of meas_astrom | It is confusing that photocal is in meas_astrom. I assume that is historical. I think it could probably live in pipe_tasks. | 2 |
| 756 | DM-1579 | Implement replacement for A.net index files | Astrometry.net index files are hard to generate and hard to read. We need another, more flexible, more standard system for storing reference files. We should also be able to read FITS files and other formats, but having a standard format with the utilities to create and query them is a must. Coming up with a format that satisfies astrometry.net's solver may be too hard, because a.net requires quads, which a non-blind solver may not need. However, a format that we can convert to something suitable for a.net would probably suffice (conversion would presumably be a separate task that is run once). It will be easier to identify a suitable format once we have identified at least one solver other than than a.net that we wish to implement or adapt. hscAstrom appears to use a catalog of star positions, which is nice and simple. | 5 |
| 757 | DM-1580 | Implement a replacement solver to the current A.net solver | This should be further refined. The solver will be required to work with several input formats. It will only be required to solve in the in the presence of a reasonable starting point with reasonable pointing errors. Failure should be graceful. If multiple, equivalent solutions are found, this should be reported (for situations including perfect grids of sources). | 15 |
| 758 | DM-1582 | Qserv spatial restrictor names are case sensitive | The SQL grammar treats Qserv spatial restrictor names case insensitively, but {{qana/QservRestrictorPlugin.cc}} does not, which means that one must use e.g. {{qserv_areaspec_box}} rather than {{qserv_areaSpec_box}}. We are loose with case in a lot of our wiki pages, so we really should fix this to avoid confusing users. Also, case insensitivity is consistent with MySQL behavior for UDF names. | 1 |
| 759 | DM-1584 | Research how to integrate different image metadata stores with DataCat | Data Release Production will generate highly structured image metadata (exposure* tables). If we decide to use DataCat (e.g., for keeping more ad hoc metadata), the question arises if/how to integrate all this together: * should we integrate all exposure* tables from all releases into DataCat? (eg via foreign tables) * should we keep them distinct, and integrate at higher level (e.g., Metadata Service) | 4 |
| 760 | DM-1585 | Design system for tracking existing images/files | We have a lot of files/images already brought in or generated through Data Challenges done to date. We need a system for cataloging them. This story will define such system, eg, sketch of UI, underlying technology used (DataCat, plain mysql, schema etc). | 6 |
| 761 | DM-1587 | Define structure of web form for collecting metadata about existing data sets | Web alpha version of the form (using django or Fermi Java webservices code) that collects input from users about data repositories. Authentication not covered in this version. | 2 |
| 762 | DM-1588 | Implement FITS header crawler and integrate it with the form | Implement crawler that walks through registered repos (through the form) and loads metadata from fits headers into the mysql backend | 7 |
| 763 | DM-1589 | Research and experiment with building form for capturing user input | Need to build a form that will be used to capture user input about existing image repositories (users will be registering their files/repositories). Options to consider: python-based django, java-based system that is part of Fermi DataCat. | 4 |
| 764 | DM-1590 | Break down & discuss DM-1074 | I will take the lead on DM-1074. First step will be to sit with [~jbosch], get a feeling for what needs to be done, and sketch out a set of stories. | 1 |
| 765 | DM-1591 | Convert test_qservAdmin.py into a real unit test | Need to turn ./admin/tests/test_qservAdmin.py into a real unit test. In the past it was broken and it went unnoticed, see DM-1395 | 1 |
| 766 | DM-1594 | Remove check for stack dir write access in qserv-configure.py | qserv-configure.py checks for write access to stack dir, this should be replace by check for read access, or removed. fjammes@qserv-build-server-xlarge:~/src/qserv$ qserv-configure.py --all INFO: Qserv configuration tool ======================================================================= WARNING : Do you want to erase all configuration data in /home/fjammes/qserv-run/2014_10.0 ? [y/n] y INFO: Copying template configuration from /home/fjammes/stack/Linux64/qserv/2014_10.0/cfg/templates to /home/fjammes/qserv-run/2014_10.0 INFO: Creating meta-configuration file: /home/fjammes/qserv-run/2014_10.0/qserv-meta.conf INFO: Reading meta-configuration file /home/fjammes/qserv-run/2014_10.0/qserv-meta.conf INFO: Defining main directory structure INFO: No write access to dir /home/fjammes/stack/Linux64/qserv/2014_10.0 : [Errno 13] Permission denied: '/home/fjammes/stack/Linux64/qserv/2014_10.0/write_tester' | 2 |
| 767 | DM-1595 | Research Javascript Frameworks: Understand Angular & React | Write some prototype code with Angular and then try to do the same thing with React | 10 |
| 768 | DM-1596 | Clean up multi-component Footprints | Following the landing of DM-1545, it's possible for an erosion operation on a footprint to cause it to split into multiple components and for peaks which were previously inside the footprint to not fall inside any of those components. Here, we should provide a "clean up" operation that takes a multiple-component footprint and splits it into a set of contiguous footprints with appropriate peak lists. | 4 |
| 769 | DM-1597 | init script fails to start xrootd after crash | I think we saw this issue in the past, not sure it was actually fixed back then or just was to reintroduced one more time. After crash of xrootd the regular {{etc/init.d/xrootd start}} fails to start it: {code} [salnikov@lsst-dbdev2 dm-621]$ ~/qserv-run/dm-621/etc/init.d/xrootd status xrootd is dead but PID file exists [FAILED] see /usr/local/home/salnikov/qserv-run/dm-621/var/run/worker/xrootd.pid [salnikov@lsst-dbdev2 dm-621]$ ~/qserv-run/dm-621/etc/init.d/xrootd start Starting xrootd: (already up) [ OK ] [salnikov@lsst-dbdev2 dm-621]$ ~/qserv-run/dm-621/etc/init.d/xrootd status xrootd is dead but PID file exists [FAILED] see /usr/local/home/salnikov/qserv-run/dm-621/var/run/worker/xrootd.pid {code} I can start it with {{restart}} but I think that {{start}} should detect that xrootd is dead ({{status}} does that) and start service correctly. This issue may exist for other services but I did not check. | 1 |
| 770 | DM-1598 | Design of calibration and ingest system | Produce a confluence page describing the approach to be taken to solve DM-1074. Ensure that all the relevant folks have reviewed that page and are happy. Break down DM-1074 into stories appropriate to that design. | 10 |
| 771 | DM-1600 | Determine if Astrometry class is desired | The question is whether the Astrometry class is the thing that is overridden or if the AstrometryTask has component configurables that are overridden. Also, determine location default implementation. | 2 |
| 772 | DM-1601 | Add support for c-style comments in front of queries sent to qserv | Connecting to qserv from java fails because the jdbc driver inserts comments. "/* ... */" in front of queries (example pasted below). The fix involves removing the comments at the proxy level {code} SQLException: Qserv error: 'ParseException:ANTLR parse error:unexpected token: /:' Query being executed when exception was thrown: /* mysql-connector-java-5.1.34 ( Revision: jess.balint@oracle.com-20141014163213-wqbwpf1ok2kvo1om ) */SHOW VARIABLES WHERE Variable_name ='language' OR Variable_name = 'net_write_timeout' OR Variable_name = 'interactive_timeout' OR Variable_name = 'wait_timeout' OR Variable_name = 'character_set_client' OR Variable_name = 'character_set_connection' OR Variable_name = 'character_set' OR Variable_name = 'character_set_server' OR Variable_name = 'tx_isolation' OR Variable_name = 'transaction_isolation' OR Variable_name = 'character_set_results' OR Variable_name = 'timezone' OR Variable_name = 'time_zone' OR Variable_name = 'system_time_zone' OR Variable_name = 'lower_case_table_names' OR Variable_name = 'max_allowed_packet' OR Variable_name = 'net_buffer_length' OR Variable_name = 'sql_mode' OR Variable_name = 'query_cache_type' OR Variable_name = 'query_cache_size' OR Variable_name = 'license' OR Variable_name = 'init_connect' {code} | 1 |
| 773 | DM-1602 | Prevent collisions in /tmp related to scisql deployment | Deploying scisql involves creating a file in /tmp. The file never gets removed. This can cause the following error when Qserv is installed later on the same machine: {code} ERROR: failed to open output file /tmp/scisql_demo_htmid10.tsv for writing {code} We should switch to using a temporary file instead of file with fixed name. | 2 |
| 774 | DM-1603 | Qserv scons scripts do not pick up the version of swig provided by eups | See the summary. The consequence is that the Qserv integration tests fail on systems that provide swig 2.x+ - there seems to be some implicit dependency on SWIG 1.x. The reason may be that swig is getting confused about shared_ptr to objects that are defined in one module, but used in another (recent swig reorganization split czar and css into two separate swig modules). | 2 |
| 775 | DM-1604 | Sanitize configs | Some solver specific information is stored in the AstrometryTask config. Further, the solver config is accessed inside the AstrometryTask run method. This mixing of information make it hard to make the framework pluggable. Solver configuration should be confined completely within the solver class (whether it be part of the Astrometry class or a configurable of its own). | 2 |
| 776 | DM-1607 | Add unit tests to test c-style comments in/around queries | I should have thought about it when doing DM-1601 but I didn't... it'd be good to add test queries that test comments before/after/inside query. | 1 |
| 777 | DM-1608 | Move meas_algorithms unit tests to meas_base framework | The following tests in meas_algorithms need to be ported to the meas_base framework: measure.py psfSelectTest.py testPsfDetermination.py ticket2019.py testCorrectFluxes.py (though this cannot be done until the algorithm exists) | 2 |
| 778 | DM-1609 | Research off-the-shelf solutions for harvesting metadata | Relevant links: * http://www.ivoa.net/documents/SIA/20141024/index.html - there’s a list of metadata in the document * CAOM (Common Archive Object Model): http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2/ | 4 |
| 779 | DM-1610 | Integrate Metadata Store prototype v1 with cat and db | Integrate prototype developed through DM-1255 with cat and db as appropriate (don't hardcode schema, don't hardcode credentials, etc). | 4 |
| 780 | DM-1611 | Experiment with DataCat foreign tables | This is a placeholder story, we should break it down into more smaller stories... | 15 |
| 781 | DM-1614 | Add support for mysql JDBC driver (v2) | MySQL client 4.1 and higher is stripping out comments before sending them to server, so the fixes done in DM-1539 are not sufficient. | 1 |
| 782 | DM-1615 | Design and implement CSS structure for distributed Qserv setup | For management of the distributed databases/tables we need info in CSS about all workers and tables. The info will be created by data loader and updated by replicator which do not exist yet. For this issue we need to provide python API which can fill the same information in CSS so that we can build and test other pieces needed for this epic. | 5 |
| 783 | DM-1616 | Implement remote host access for management framework | To manage remote workers we need a way to access services on remote machines that run workers. Services may be something like mysql (which we would prefer to run without TCP port open) and optionally xrootd. This ticket will implement Python module which will hide a complexity of doing things like ssh/port forwarding/authentication from the client. | 5 |
| 784 | DM-1617 | Client library for accessing distributed database/table information from CSS | Provide Python interface for accessing information in CSS which is relevant to distributed management, such as database/table/node data. This interface can be used also by data loader and replicator. | 8 |
| 785 | DM-1618 | Implement distributed database creation | Implement Python library which creates databases on all active workers based on info from CSS. | 10 |
| 786 | DM-1619 | Implement distributed table creation | Implement Python library for creating mysql tables on all active workers. | 11 |
| 787 | DM-1620 | Move CSS documentation close to code | CSS documentation about the structure is currently in trac (which is readonly), at https://dev.lsstcorp.org/trac/wiki/db/Qserv/CSS. We need to move it close to the source code, e.g., to a doc file. | 2 |
| 788 | DM-1621 | Add unit test to to verify zookeeper versioning works | This is a follow u pto DM-1453, we need to add a unit test that will prove that mismatched versions in zookeeper and software are properly handled. | 1 |
| 789 | DM-1622 | Add unit test for queries from DM-1539 | Need to add unit test for queries listed in DM-1539 | 1 |
| 790 | DM-1624 | Qserv should report it's version | It should be possible to determine which version of qserv we are running by looking at information from log files. So, in practice, we probably should generate in scons a unique id (from sha from git?) and compile it into the code. | 1 |
| 791 | DM-1625 | SciSQL should report its version | It should be possible to determine which version of scisql we are running. | 1 |
| 792 | DM-1626 | Build 2014_12 Qserv release | The title says it all. Please open ticket for next release when closing this one. | 1 |
| 793 | DM-1627 | Integrate metaserv, imgserv and dbserv with Data Access Services | * Create dbserv for handling database related web-based queries, support running queries through web (initial version). * Integrate dbserv, metaserv and imgserv RESTful API into the webserv - want to run all services (meta, image, db) under one server. * Also want to be able to run them separately, so have some handy servers for each service * Proof of concept for supporting multiple formats (html, json) | 22 |
| 794 | DM-1629 | Adopted/Retired RFCs are not counted as resolved | Also, marking an RFC as Adopted brings up a box with a message applicable to the Closed status. | 1 |
| 795 | DM-1630 | New RFCs should result in dm-devel E-mails and HipChat postings | Email notices of new RFCs filed with a DM component go to dm-devel Email notices of new RFCs filed with a DM component go to Data Management chat room Email notices of all new RFCs go to Bot: RFC room | 1 |
| 796 | DM-1632 | Build 2014_11 Qserv release | The title says it all. Please open ticket for next release when closing this one. Tasks to do: - publish last buildbot build under a temporary eups-tag ("qserv-dev")and test it, if it works fine: - create git-tags for Qserv and dependencies - publish the release with eups-tags "qserv" and "YYYY_MM" - generate and publish the doc for release "YYYY_MM" - update release number in Qserv code and set "YYYY_MM+1" as release in dev and "YYYY_MM" as stable release (update {{admin/bin/qserv-version.sh}}, {{doc/source/conf.py}}, {{doc/source/toplevel.rst}}) - generate and publish the doc for release "YYYY_MM+1" - look at the doc - commit {{admin/bin/qserv-version.sh}}, {{doc/source/conf.py}}, {{doc/source/toplevel.rst}} with current ticket number This procedure should be validated and documented. | 1 |
| 797 | DM-1633 | Update build process for Firefly opensource | Update build and deploy related scripts to reflects the changes affected by open sourcing of Firefly. | 7 |
| 798 | DM-1635 | Remove redundant CORS headers from Firefly's http response | Make sure CORS related headers are not sent when the Origin header is missing. Firefox does not like it. | 1 |
| 799 | DM-1636 | Research popular web development technologies | Research popular web development technologies to prepare Firefly for the future with the focus on front-end framework. | 6 |
| 800 | DM-1641 | Document "getting started" procedure for new stack developers | Document a procedure for building the stack on a new system in a way that is appropriate for both project members and external contributors. This can be linked as a "getting started" guide from http://dm.lsst.org/. | 5 |
| 801 | DM-1642 | LOE - Week ending 12/5/14 | NULL | 8 |
| 802 | DM-1645 | preparation work for FIrefly open source | 1. discuss with IPAC director and managers to open source Firefly 2. study the major open source license,Apache, GPL, BSD 3-clause, MIT. 3. final decision: BSD-3 clause | 6 |
| 803 | DM-1647 | Use an OpenStack instance to run an HTCondor Central manager | Use an OpenStack instance to run an HTCondor Central manager | 4 |
| 804 | DM-1648 | W16 Research technologies for Data Access | Research technologies potentially useful for Data Access / Database. This epic covers Apache Mesos, Google Kubernetes, MaxScale, Serf, Consul, and MemSQL. | 38 |
| 805 | DM-1652 | The existing FITS reader class needs to be refactored to improve the performance(1) | - checkout the classes - understand the FITS standard and current implementation | 16 |
| 806 | DM-1653 | Extend data loading script to support multi-node setup | Implement simplest use case for data loading with one or more worker database separate from czar database. Simplest means minimal functionality in what concerns access to workers, just assume for now that we can connect to every worker directly using regular TCP connection. It should be possible to just add a list of worker nodes as an option to loader script and send the chunks in some random order to the workers in that list. Of course chunks for different tables should end up on the same host, so some form of chunk management is needed (use for now whatever is defined in CSS doc on trac). | 8 |
| 807 | DM-1655 | Working with SLAC on definition of metadata store | Follow up metadata store schema development to ensure SUI will be able to use it. Define the fields that should go into Data Definition table. Define the fields that must be present in the image metadata table, which SUI will be searching. | 2 |
| 808 | DM-1656 | hipchat support for maigically urlifying docushare documents | It would be generally use full if references to official documents in hipchat (or it's successor) automagically generated urls for official document handles. Eg: Document-1234, LSE-123 | 1 |
| 809 | DM-1657 | LOE - Week ending 12/12/14 | NULL | 8 |
| 810 | DM-1658 | Add git bisect tool for Qserv repos | {code:bash} fjammes@clrlsst-dbmaster-vm:~/src/qserv (u/fjammes/DM-627 $%) $ qserv-test-head.sh -h Usage: qserv-test-head.sh [options] Available options: -h this message -q quick: only rebuild/install new Qserv code, and perform test case #01 Rebuild from scratch, configure and run integration tests against a Qserv git repository. Pre-requisite: source loadLSST.bash setup qserv_distrib -t qserv setup -k -r ${QSERV_SRC_DIR} Can be used with 'git bisect' : cd ${QSERV_SRC_DIR} git bisect start git bisect bad git bisect good git-commit-id git bisect run /home/fjammes/src/qserv_testdata/bin/qserv-test-head.sh {code} Code is in DM-627 ticket branch. | 2 |
| 811 | DM-1659 | Aperture Flux back into an abstract class | The last change to ApertureFlux to make it work with the new C++ design changed ApertureFluxAlgorithm into an instantiatable class. However, I have now figured out how to make this work with SWIG while still allowing measure and fail to be defined by default at the ApertureFlux level.. So this issue is to put things back in order. | 1 |
| 812 | DM-1660 | Statistics tests use a constant image | I just noticed that the tests for Statistics (clipped mean etc.) use a constant image. We should be testing a Gaussian field (although that makes the tests a little trickier) | 2 |
| 813 | DM-1661 | czar log4cxx link/load bug | Under ubuntu 14.04 (at least), the czar falls over at load time with an unresolved sym for typeinfo for log4cxx::helpers::ObjectPtrBase while loading the css python wrapper shared lib. | 2 |
| 814 | DM-1662 | Make qserv dependencies build on OS X with clang | Fix anything necessary for qserv dependencies to build on OS X with clang. Note -- making qserv itself build is more complicated and may require a separate ticket. | 4 |
| 815 | DM-1663 | fix dependency problems in obs_subaru scons scripts | When building obs_subaru with -j4, it often tries to build files related to the defects before the main Python module is built, resulting in import errors (because the scripts it invokes depend on the main Python module). We need to rewrite the SCons scripts to ensure this dependency is captured. A preliminary look indicated that this is not entirely trivial, and I'll have to remind myself a bit of how some things in SCons work to get it done, so I'm putting this off for a future sprint. In the meantime, the workaround is to build obs_subaru with no parallelization. | 2 |
| 816 | DM-1664 | Understand historical written docushare materials dealing with the operational security environment. | researched the docushare traversing the plans for materials to be embedded in the OCS and similar systems, as well as extant operational plans. The goal was to understand how to separate the the security responsibilities of development, what security constraints ought to be but on items that are delivered, and what to tell the camera and telecscope teams to mender ea smooth integration with IT security systems upon delivey and integration of their sub systems in Chile. | 3 |
| 817 | DM-1667 | Install PgMySQL and use to connect to local Qserv. | Used "pip" to install it. "Conda" should work as well. Therefore, it should be easy to make it part of the delivered system: VM, container, tar file, after the fact download, etc. It has documentation, uses the MIT license, under active development and available from PyPI. DB connection is straight forward and requires little experience to get meaningful work done. | 2 |
| 818 | DM-1668 | Create SQL code to read Qserv into Python Pandas data frame | This works well, at least for a simple case. You can move directly from a query statement to a Pandas data frame for analysis in just a few lines of code. Here is the start of an iPython Qserv session showing how easy it is. In [6]: import pandas as pd In [7]: import pymysql as db In [8]: conn = db.connect(host='lsst-db1.ipac.caltech.edu',port=4040, user='qsmaster', passwd='', db='LSST') In [11]: df = pd.read_sql("select deepCoaddId, tract, patch, ra, decl from DeepCoadd", conn) In [12]: df Out[12]: deepCoaddId tract patch ra decl 0 26607706 0 406,11 0.669945 1.152218 1 26673242 0 407,11 0.449945 1.152218 2 26804242 0 409,2 0.011595 -0.734160 3 26673154 0 407,0 0.449945 -1.152108 … | 2 |
| 819 | DM-1669 | Explore queries for Qserv database. | Use a local database here at IPAC with 5 Qserv tables in it. Looked at several Python query interfaces. Used the pymysql interface for testing because it's pure Python and because I found reports suggesting it was almost as fast as the mysqldb interface that requires C language libraries. Ad hoc queries can be constructed in three lines of code, so useable in a science environment. Found a couple of bugs in Qserv that were reported. | 6 |
| 820 | DM-1670 | Begin looking at how Python Pandas can be used for LSST data analysis. | Pandas is well integrated with the other parts of SciPy: numpy, matlibpy, etc. It’s a good candidate for data analysis, especially where time series are involved. However, there are no multidimensional columns, poor metadata support for FITS files and a need to use masks instead of NaN values. These may, or may not, be problems. There is a 400 page book about Pandas, so it will take some further time to learn its value, especially with astronomical data in different situations. | 5 |
| 821 | DM-1673 | Allow SWIG override for broken SWIG installations | Dependency on SWIG 2.0+ was introduced into Qserv, and this broke Qserv building on systems relying on SWIG 1.3.x. This ticket introduces basic code to override SWIG_LIB on those systems to allow use of the broken installation (some SWIG search paths are fixed during its build process otherwise). | 1 |
| 822 | DM-1674 | Data ingest scripts cleanup | NULL | 10 |
| 823 | DM-1675 | Identify dead code | NULL | 1 |
| 824 | DM-1676 | Write documentation for SSI interface | NULL | 5 |
| 825 | DM-1677 | multi-error utility class | QueryAction::Impl currently has a handful of private members/methods related to maintaining a collection of errors that occurs while processing a query (_addErrorMsg, IntString, IntStringVectory, _errors, etc.) It would be useful to split this stuff out into a separate, re-usable utility class, and extend, then extend with some additional functionality (output stream operator, subclass from std::exception, maybe capture __LINE__ and __FILE__, integrate with logging, etc.) [Fabrice: discuss requirements and basic design w/ Fritz and/or Daniel] | 10 |
| 826 | DM-1685 | Minor bug in a test | tests/centroid.py has a bug in testMeasureCentroid: "c" is undefined in the following bit of code: {code} if display: ds9.dot("x", c.getX(), c.getY(), ctype=ds9.GREEN) {code} | 1 |
| 827 | DM-1692 | Implement "unlimited" result size handling | DM-854 exposed an issue in handling large results. Result rows are returned from worker to czar in protobufs messages. However, protobufs messages should not be larger than some number of megabytes, according to protobufs documentation. IOW, protobufs is not designed to handle messages on the order of hundreds of megabytes. There may be some code in the protobufs implementation that does not scale beyond messages of a few megabytes. Hence, in order to send larger amounts of result rows, we need to use multiple messages. The current protobufs definition for result messages includes a placeholder for chaining result messages, but there is no code on the czar or worker that implements result chaining. The scope of this ticket is to implement message chaining for results on the worker and on the czar, in such a way that it places no limits on the overall number of result rows (if there is a limit, it should be no smaller than quadrillions and be well-documented). | 15 |
| 828 | DM-1694 | Define interfaces for Data Access Services | NULL | 8 |
| 829 | DM-1695 | Implement interfaces for Data Access Services | Implement proof of concept, skeleton of the prototype. The work will continue in follow up stories in February and in S15. | 8 |
| 830 | DM-1696 | Integrate image cutout service interfaces with butler | NULL | 5 |
| 831 | DM-1697 | Finish image cutout service implementation | Define appropriate interfaces and connect them with the RESTful API (see DM-1695). | 9 |
| 832 | DM-1698 | Finish metadata store prototype | * A quick proof-of-concept prototype of loading tool for loading database-information into metadata store. * Handling mysql credentials through auth file in home dir instead of hardcoded values. | 4 |
| 833 | DM-1700 | Create read only OpenStack volume and execute processing scenario | Create read only OpenStack volume and execute processing scenario | 4 |
| 834 | DM-1701 | Clone OpenStack volume for use against multiple instances | Clone OpenStack volume for use against multiple instances | 2 |
| 835 | DM-1702 | Examine fqdn/hostname assignment for OpenStack instance | Examine fqdn/hostname assignment for OpenStack instance | 4 |
| 836 | DM-1703 | S15 Implement Database & Table Mgmt | Continuation of DM-1036, making the code for managing distributed databases and tables more feature reach, including features such as deletion. | 53 |
| 837 | DM-1704 | S15 Run Large Scale Qserv Tests | Run large scale tests to uncover unexpected issues and bottlenecks. | 48 |
| 838 | DM-1705 | S15 Tune Qserv | Fix scalability and performance issues uncovered through large scale tests DM-1704 | 100 |
| 839 | DM-1706 | S15 Analyze Qserv Performance | Final analysis of Qserv performance, measure KPIs. Based on LDM-240, we are aiming to demonstrate: * 50 simultaneous low volume queries, 18 sec/query * 5 simultaneous high-volume queries, 24 h/query * data size: 10% of DR1 level. * Continuous running for 24 h with no software failures. | 5 |
| 840 | DM-1707 | S15 Refactor Qserv | Ongoing refactoring of Qserv - code cleanup, tightening interfaces etc. | 100 |
| 841 | DM-1708 | W16 Improve Query Coverage in Qserv | Currently Qserv supports only a limited subset of queries. We need to make sure it supports all queries that users need to run. | 46 |
| 842 | DM-1709 | Implement result sorting for integration tests | We need to be able to sort results, because we can't always rely on ORDER BY. So we need a formatting per query in the integration tests (sort result for some, don't sort for others etc.) The following queries have been disabled because we don't have result sorting, so once it is implemented, we will need to re-enabled them prior to closing this ticket: {code} case02/queries/0003_selectMetadataForOneGalaxy_withUSING.sql case02/queries/3001_query_035.sql case02/queries/3008_selectObjectWithColorMagnitudeGreaterThan.sql case02/queries/3011_selectObjectWithMagnitudes.sql case02/queries/3011_selectObjectWithMagnitudes_noalias.sql {code} | 8 |
| 843 | DM-1710 | ValueError in lsst.afw.table.Catalog.extend() | {code} from lsst.afw.table import BaseCatalog, Schema s = Schema() c1 = BaseCatalog(s) c2 = BaseCatalog(s) c1.extend(c2) {code} The above fails, saying: {code} Traceback (most recent call last): File "test.py", line 7, in <module> c1.extend(c2) File "/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/afw/10.0+3/python/lsst/afw/table/tableLib.py", line 6909, in extend _tableLib.BaseCatalog_extend(self, iterable, deep) ValueError: invalid null reference in method 'BaseCatalog_extend', argument 3 of type 'lsst::afw::table::SchemaMapper const &' {code} | 1 |
| 844 | DM-1711 | S15 Improve MetaServ: RESTful, Basic Image Search, DDL, Config Files | Implement beta version of the Metadata Service. This version will support basic image search, DDL, config files, and RESTful interfaces for MetaServ and Database (Qserv). | 57 |
| 845 | DM-1712 | S15 Add Support for Image Stitching and Rotation, RESTful APIs | Implement image stitching and rotating, including RESTful APIs. | 35 |
| 846 | DM-1713 | S15 Image & File Archive v2 | System for tracking existing image data sets integrated with metadata services. | 5 |
| 847 | DM-1714 | Integrate MetaServ with Schema Browser | Schema browser displays detailed info about schema, including custom fields like UCD, units etc. This information is stored as comments embedded in the master version of the schema (in "cat" repo). Currently we are generating ascii from the master schema for schema browser, and we load it into mysql, then schema browser reads it from mysql. This story involves changing schema browser such that it will read the schema information directly from MetaServ. | 6 |
| 848 | DM-1715 | Disable query killing | Apparently killing a query through Ctrl-C is confusing xrootd. Disable query killing (which seems to be only partly implemented). | 1 |
| 849 | DM-1716 | Implement query killing through Ctrl-C | Need to properly implement query killing through Ctrl-C | 16 |
| 850 | DM-1720 | Make secondary index for director table only | Following discussion on qserv-l, we only need to generate "secondary" index for director table, no other table is supposed to have it. Need to modify data loader to recognize which table is director table and generate index only for that table. | 2 |
| 851 | DM-1721 | S15 Improve Query Coverage in Qserv | Query coverage in the qserv integration testing is very limited, we have been turning off more and more queries and we were making the qserv code and the data loader more strict. This epic covers work (fixes and improvements) related to * re-enabling test queries marked as "fixme" (when it make sense, some queries are for features that are not implemented yet) * adding more queries to test interfaces and features that are implemented but are not currently tested. | 40 |
| 852 | DM-1722 | LOE - Week ending 12/19/14 | The System Administration team at NCSA worked on the following LOE tasks this week: - RMA'ed RAID card for lsst-dev <https://jira.ncsa.illinois.edu/browse/LSST-599> - Updated user's updated public ssh keys <https://jira.ncsa.illinois.edu/browse/LSST-614> - Rebuilt failed drive in lsst-dbdev2 <https://jira.ncsa.illinois.edu/browse/LSST-610> - Increasing drive size of lsst-eval <https://jira.ncsa.illinois.edu/browse/LSST-611> - Researched & repaired lsst20 NFS issues <https://jira.ncsa.illinois.edu/browse/LSST-591> - Upgraded lsst-xfer to 10G networking | 14 |
| 853 | DM-1725 | stack build fails on gcc 4.8 with opt=3 | The stack fails to build with gcc 4.8, with a test failure in meas_base (though a similar problem on the HSC fork suggests the problem is actually in afw). [~darko] reports that on OpenSuse 13.1, the failure goes away when compiling with opt=1 instead of the default opt=3, indicating that the problem is overly aggressive optimization. This, and the fact that a traceback of the HSC-side failure implicates MaskedImage::getXY0, leads me to guess that something is going wrong with the alignment of the Eigen data members in afw::geom::Point. Until that theory is disproven, this probably belongs in my court, though I'd be happy to let someone steal it from me if they're interested in working on it. | 4 |
| 854 | DM-1726 | Stabilize Firefly Repository | - Make GitHub firefly ready but not public - Add firefly to LSST git - Clean up any lingering issues from separation - break up IRSA Viewer from Firefly Viewer | 14 |
| 855 | DM-1727 | Research Javascript Frameworks: Work toward future proposal | Take prior research and come up with rough firefly migration proposal. Include how to use a hybrid system for foreseeable future. Write some prototype code. | 16 |
| 856 | DM-1731 | fix table file handling of MANPATH in dependencies | As discussed on DM-1220, the table files for: - mysqlproxy - protobuf - lua - expat should have the MANPATH entry removed entirely, while: - xrootd should have ":" added to the end of its MANPATH value, to allow the default paths to be searched as well. | 1 |
| 857 | DM-1732 | Fix error on duplicate result_id_m table while launching qserv integration tests | Next command: {code} qserv-check-integration.py --case=01 --load; qserv-check-integration.py --case=02 --load {code} Fails, most of the time, with next error in logs: {code} qserv@clrinfoport09:~/src/qserv (u/fjammes/DM-627 *+)⟫ cat ~/qserv-run/2014_12/var/log/qserv-czar.log | grep result_1211906_m 0103 16:38:18.576 [0x7f23c4ff9700] DEBUG root (build/rproc/InfileMerger.cc:432) - InfileMerger create table:CREATE TABLE qservResult.result_1211906_m (`ra` DOUBLE) 0103 16:38:18.661 [0x7f23c4ff9700] DEBUG root (build/rproc/InfileMerger.cc:354) - InfileMerger sql success: CREATE TABLE qservResult.result_1211906_m (`ra` DOUBLE) 0103 16:38:18.661 [0x7f23c4ff9700] DEBUG root (build/rproc/InfileMerger.cc:445) - InfileMerger table qservResult.result_1211906_m is ready 0103 16:38:18.686 [0x7f23c77fe700] INFO root (build/rproc/InfileMerger.cc:299) - Merging w/CREATE TABLE qservResult.result_1211906 SELECT ra AS ra FROM qservResult.result_1211906_m ORDER BY ra 0103 16:38:18.720 [0x7f23c77fe700] DEBUG root (build/rproc/InfileMerger.cc:354) - InfileMerger sql success: CREATE TABLE qservResult.result_1211906 SELECT ra AS ra FROM qservResult.result_1211906_m ORDER BY ra 0103 16:38:18.720 [0x7f23c77fe700] INFO root (build/rproc/InfileMerger.cc:305) - Cleaning up qservResult.result_1211906_m 0103 16:38:18.721 [0x7f23c77fe700] INFO root (build/rproc/InfileMerger.cc:317) - Merged qservResult.result_1211906_m into qservResult.result_1211906 0103 16:38:24.716 [0x7fe3cf7fe700] DEBUG root (build/rproc/InfileMerger.cc:432) - InfileMerger create table:CREATE TABLE qservResult.result_1211906_m (`QS1_COUNT` BIGINT(21)) 0103 16:38:24.717 [0x7fe3cf7fe700] ERROR root (build/rproc/InfileMerger.cc:351) - InfileMerger sql error: Error applying sql. Error 1050: Table 'result_1211906_m' already exists Unable to execute query: CREATE TABLE qservResult.result_1211906_m (`QS1_COUNT` BIGINT(21)) 0103 16:38:24.717 [0x7fe3cf7fe700] ERROR root (build/rproc/InfileMerger.cc:438) - InfileMerger error: Error creating table (qservResult.result_1211906_m) 0103 16:38:34.672 [0x7fe3deffd700] INFO root (build/rproc/InfileMerger.cc:299) - Merging w/CREATE TABLE qservResult.result_1211906 SELECT SUM(QS1_COUNT) AS OBJ_COUNT FROM qservResult.result_1211906_m 0103 16:38:34.673 [0x7fe3deffd700] ERROR root (build/rproc/InfileMerger.cc:351) - InfileMerger sql error: Error applying sql. Error 1054: Unknown column 'QS1_COUNT' in 'field list' Unable to execute query: CREATE TABLE qservResult.result_1211906 SELECT SUM(QS1_COUNT) AS OBJ_COUNT FROM qservResult.result_1211906_m 0103 16:38:34.673 [0x7fe3deffd700] INFO root (build/rproc/InfileMerger.cc:305) - Cleaning up qservResult.result_1211906_m 0103 16:38:34.674 [0x7fe3deffd700] INFO root (build/rproc/InfileMerger.cc:317) - Merged qservResult.result_1211906_m into qservResult.result_1211906 {code} So it seems test case #01 create table .result_1211906_m, and then test case #02 try to re-use this name for an other query. Qserv result tables aren't cleaned (here Qserv has be stopped): {code} ls ~/qserv-run/2014_12/var/lib/mysql/qservResult/ db.opt result_1211336_m.MYI result_1211340_m.MYI result_1211343_m.MYI result_1211346_m.MYI result_1211382_m.MYI result_1211385_m.MYI result_1211902_m.MYI result_1211905_m.MYI result_1211334_m.frm result_1211337_m.frm result_1211341_m.frm result_1211344_m.frm result_1211347_m.frm result_1211383_m.frm result_1211386_m.frm result_1211903_m.frm result_1211906_m.frm result_1211334_m.MYD result_1211337_m.MYD result_1211341_m.MYD result_1211344_m.MYD result_1211347_m.MYD result_1211383_m.MYD result_1211386_m.MYD result_1211903_m.MYD result_1211906_m.MYD result_1211334_m.MYI result_1211337_m.MYI result_1211341_m.MYI result_1211344_m.MYI result_1211347_m.MYI result_1211383_m.MYI result_1211386_m.MYI result_1211903_m.MYI result_1211906_m.MYI result_1211335_m.frm result_1211339_m.frm result_1211342_m.frm result_1211345_m.frm result_1211381_m.frm result_1211384_m.frm result_1211901_m.frm result_1211904_m.frm result_1211335_m.MYD result_1211339_m.MYD result_1211342_m.MYD result_1211345_m.MYD result_1211381_m.MYD result_1211384_m.MYD result_1211901_m.MYD result_1211904_m.MYD result_1211335_m.MYI result_1211339_m.MYI result_1211342_m.MYI result_1211345_m.MYI result_1211381_m.MYI result_1211384_m.MYI result_1211901_m.MYI result_1211904_m.MYI result_1211336_m.frm result_1211340_m.frm result_1211343_m.frm result_1211346_m.frm result_1211382_m.frm result_1211385_m.frm result_1211902_m.frm result_1211905_m.frm result_1211336_m.MYD result_1211340_m.MYD result_1211343_m.MYD result_1211346_m.MYD result_1211382_m.MYD result_1211385_m.MYD result_1211902_m.MYD result_1211905_m.MYD {code} Changing _idCounter to next value in appInterface.py: {code:python} self._idCounter = int(time.time() % (60*60*24*365) * 10) {code} solves the problem, but a deeper explanation will allow to bring a more robust fix. | 8 |
| 858 | DM-1733 | Build 2015_01 Qserv release | See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe. | 1 |
| 859 | DM-1734 | refactor fftools viewer to have derived viewers / start LSST SUI git repo | Do the following: - refactor fftools to allow for specialized viewers. - make a generic fftools viewer. - make an irsaviewer package in ife-new that will build. - generalize the catalog search and add factories to access the lsst search process. | 10 |
| 860 | DM-1735 | Have newinstall.sh check itself against distrib version | We want to alert people who are just using a newinstall.sh they have lying around (old or hacked up or...) that they are not using the official server version. | 1 |
| 861 | DM-1736 | Migrate qserv modules (python code) to new logging system | NULL | 8 |
| 862 | DM-1738 | deblender artifacts in noise-replaced images | We still see noise artifacts in some deblended images on the LSST side when running the M31 HSC data. They look like the result of running NoiseReplacer on HeavyFootprints in which the children can extend beyond the parents. This was fixed on the HSC side on DM-340 (before the HSC JIRA split off), and I *think* we just need to transfer the fix to LSST. | 1 |
| 863 | DM-1739 | Catch-all epic for essential fixes during DM-W15-4 | NULL | 10 |
| 864 | DM-1743 | CSV reader for Qserv partitioner doesn't handle no-escape and no-quote options properly | Both the no-quote and no-escape CSV formatting command line options should not have a default value, as specifying any value turns off field escaping and quoting. Furthermore, when quoting is turned off, the reader incorrectly treats embedded NUL characters as a quote character. | 1 |
| 865 | DM-1744 | Fix SWIG_SWIG_LIB empty list default value | See Serge message to Qserv-l "xrootd premature death": {quote} However, there are bigger problems. First of all, master doesn’t build for me. I get this error: File "/home/lsstadm/qserv/SConstruct", line 104: env.Alias("dist-core", get_install_targets()) File "/home/lsstadm/qserv/SConstruct", line 90: exports=['env', 'ARGUMENTS']) File "/home/lsstadm/stack/Linux64/scons/2.3.0+1/lib/scons/SCons/Script/SConscript.py", line 609: return method(*args, **kw) File "/home/lsstadm/stack/Linux64/scons/2.3.0+1/lib/scons/SCons/Script/SConscript.py", line 546: return _SConscript(self.fs, *files, **subst_kw) File "/home/lsstadm/stack/Linux64/scons/2.3.0+1/lib/scons/SCons/Script/SConscript.py", line 260: exec _file_ in call_stack[-1].globals File "/home/lsstadm/qserv/build/SConscript", line 39: canBuild = detect.checkMySql(env) and detect.setXrootd(env) and detect.checkXrootdLink(env) File "/home/lsstadm/qserv/site_scons/detect.py", line 225: xrdLibPath = findXrootdLibPath("XrdCl", env["LIBPATH"]) File "/home/lsstadm/qserv/site_scons/detect.py", line 213: if os.access(os.path.join(path, fName), os.R_OK): File "/home/lsstadm/stack/Linux64/anaconda/2.1.0/lib/python2.7/posixpath.py", line 77: elif path == '' or path.endswith('/'): which is caused by the fact that env[“LIBPATH”] looks like: [[], '/home/lsstadm/stack/Linux64/antlr/2.7.7/lib', '/home/lsstadm/stack/Linux64/boost/1.55.0.1.lsst2/lib', '/home/lsstadm/stack/Linux64/log4cxx/0.10.0.lsst1+2/lib', '/home/lsstadm/stack/Linux64/xrootd/4.0.0rc4-qsClient2/lib', '/home/lsstadm/stack/Linux64/zookeeper/3.4.6/c-binding/lib', '/home/lsstadm/stack/Linux64/mysql/5.1.65.lsst1/lib', '/home/lsstadm/stack/Linux64/protobuf/2.4.1/lib', '/home/lsstadm/stack/Linux64/log/10.0+3/lib'] The first element is [], which comes from https://github.com/LSST/qserv/blob/master/site_scons/state.py#L173 where a PathVariable called SWIG_SWIG_LIB is given a default value of []. I can fix the build by changing the default to an empty string… but I don’t know enough scons to say whether that’s the right thing to do. Can one of the scons gurus confirm that’s the right fix? {quote} | 1 |
| 866 | DM-1753 | make lsst-sui & firefly git repo / make an lsst viewer | NULL | 4 |
| 867 | DM-1754 | Update auto build tool to work with new split repositories | After the repository split, changes are required to get the auto build tool to work properly. Firefly and Firefly based applications are built using Gradle system. | 8 |
| 868 | DM-1755 | Create an integration test case with GB-sized data | It's difficult to load manually data in Qserv, so a way to do that is to use integration test framework to automatically do this. Big data file won't be stored in git, but the user wil lhave to retrieve them manually, and the test case won't be executed by integration tests. | 4 |
| 869 | DM-1761 | Provide input data for exampleCmdLineTask.py | {{pipe_tasks/examples/exampleCmdLineTask.py}} reads data from a repository. The comments in {{pipe_tasks/python/lsst/pipe/tasks/exampleCmdLineTask.py}} suggest that {code} # The following will work on an NCSA lsst* computer: examples/exampleCmdLineTask.py /lsst8/krughoff/diffim_data/sparse_diffim_output_v7_2 --id visit=6866601 {code} There are a few problems with that: * External contributors don't have access to {{lsst*}}; * Even though that data exists now, it's unclear how long it will remain there, or what steps are being taken to preserve it; * The mention of this data is fairly well buried -- it does appear in the documentation, but it's certainly not the first thing a new user will stumble upon. At least the first two points could be addressed by referring to a publicly available data repository. For example, the following works once {{afwdata}} has been set up: {code} examples/exampleCmdLineTask.py ${AFWDATA_DIR}/ImSim --id visit=85408556 {code} Although this has the downside of only providing a single image. | 1 |
| 870 | DM-1762 | Export SUI data (DC_W13_Stripe82_subset) | - import sui.sql.bzip2.out (produced by Serge) into MySQL for DeepSource and DeepForcedSource tables: - remove columns chunkId and subChunkId for each chunk table - merge all chunk table into the main table - join DeepSource and DeepForcedSource to add coordinates of DeepSource (director) object in DeepForcedSource table. then dump DeepSource and DeepForcedSource to files DeepSource.csv and DeepForcedSource.csv {code:sql} SELECT f.*, COALESCE(s.ra, f.ra), COALESCE(s.decl, f.decl) FROM DeepForcedSource f LEFT JOIN DeepSource s ON (f.deepSourceId = s.deepSourceId) INTO OUTFILE '/db1/dump/DeepForcedSource.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n'; {code} - Load this file using Qserv loader. A sample should be made and tested first to validate this procedure. This sample could be added in qserv_testdata | 3 |
| 871 | DM-1764 | overhaul slot and alias system | While working on DM-1218 and DM-464, I've grown quite dissatisfied with the current state of the slot and alias mechanisms, and we now have a concrete proposal for larger-scale changes on RFC-11. Unfortunately, I don't think we'll be in a good position to do much about this until we've completed the transition to meas_base and removed the old measurement framework in meas_algorithms. | 6 |
| 872 | DM-1765 | move SourceRecord/Table/Catalog to meas_base | We can make address a lot of dependency issues if we move the Source classes to meas_base, because we'll no longer have low-level code (e.g. afw::table persistence) in the same module as very high-level code (e.g. slots). It will also put all the slot code in the same place, instead of spreading it across two pacakges. This should be straightforward, except that we'll have a lot of downstream code to (trivially) change, and there's a good chance Swig will get confused somewhere along the way. | 2 |
| 873 | DM-1766 | Remove in-memory support of old-version afw::table objects | After removing the old measurement framework in meas_algorithms, we should also end support for version=0 Schemas in memory, and instead convert version=0 Schemas to version 1 when we unpersist them. | 11 |
| 874 | DM-1767 | afw::table - post-transition improvements | Breakdown: jbosch 40%; swinbank 60% | 23 |
| 875 | DM-1769 | Measurement - Framework Improvements | Add new features in meas_base that are desirable, but not required to replace functionality in meas_algorithms. Breakdown: pgee 70%; jbosch 30% | 60 |
| 876 | DM-1770 | Support DDL in MetaServ - design | DDL information is embedded as comments in the master version of the schema (in "cat" repo). Currently we are only using it for schema browser. This story involves designing the procedure involving loading DDL information into MetaServ. We need to be ready to support a variety of scenarios: * we are getting already preloaded database, need to just load metadata about it to metaserv (we might have the original ascii file with extra information, or not) * we are starting from scratch, need to initialize database (including loading schema), and need to load the information to the metaserv * we already have the database and metadata in metaserv, but we want to change something (eg. alter table, or delete table, or delete database). | 2 |
| 877 | DM-1771 | move executionOrder from plugin config class to plugin class | We originally put the executionOrder parameter (which determines when a plugin is run, relative to others), in the config object, simply because that's where it was in the old framework. But it's really not something that should be configurable, as it depends only on the inputs the algorithm needs, which don't change. | 1 |
| 878 | DM-1772 | Create a search processor to do cone/box search on a QSERV catalog | Create a search processor which accepts cone and box spatial constraints, queries a catalog, stored on QSERV (DeepSource), and returns the relevant rows as an IPAC table or RawDataSet. The implementation is querying DeepSource catalog using qserv_areaspec_circle and qserv_areaspec_box spatial restrictors. | 11 |
| 879 | DM-1773 | Read SUI requirements, send a list of questions to the group scientist | Read the requirements document on Archive Browser and Query Tools, make a record of unclear items, send questions to our scientist (D. Ciardi) | 5 |
| 880 | DM-1774 | Add support for running unit tests in qserv/admin | This came up during review of DM-370: "We do not run tests scripts in admin/ during regular build, SConscript in admin/ does not support that unfortunately." This story involves tweaking SConscript to enable running unit tests automatically. | 1 |
| 881 | DM-1775 | Setup work environment / test builds for refactored repositories | After the repository was split into firefly and new-ife, it was necessary to understand the changes, check for inconsistencies, test builds, set up new IDEA project, etc. | 3 |
| 882 | DM-1776 | Obtain and use catalog dd (data definition) | Get catalog metadata, which should include column description, units, and type. Use it in tool tips and possibly to create flexible constraints. A temporary solution is to get metadata from an internal lsst_schema_browser_S12_lsstsim database on lsst-db.ncsa.illinois.edu A permanent solution would be querying catalog metadata from metadata store. | 10 |
| 883 | DM-1777 | XYPlotter should be caching and restoring plot metadata | For efficiency, XYPlotter is designed to create up to 4 cards. When the card limit is exceeded, a previously created card is reused to plot catalog data for the current catalog. When card is reused the previous plot metadata (like column selections, grid option, etc.) are lost. | 4 |
| 884 | DM-1778 | Remove optimisation flag management in partition SConstruct | - eups build with -g and -O3 by default. - Developpers can build their sources with next options: scons debug=false opt=3 - Nevetheless partition SConstruct add -O2 option if debug is disabled, which is orthogonal. | 2 |
| 885 | DM-1782 | The existing FITS reader class needs to be refactored to improve the performance(2) | - FITS reader class refactoring, improve readability - validation of the new code against the old one - unit test - performance improvement recording if possible | 16 |
| 886 | DM-1783 | fix faint source and minimum-radius problems in Kron photometry | This transfers some improvements to the Kron photometry from the HSC side: - HSC-983: address failures on faint sources - HSC-989: fix the minimum radius - HSC-865: switch to determinant radius instead of semimajor axis - HSC-962: bad radius flag was not being used - HSC-121: fix scaling in forced photometry The story points estimate here is 50% of the actual effort, as the work (already done) also benefited HSC. | 5 |
| 887 | DM-1784 | Fix errors in parsing or rendering nested expressions | Qserv has errors rendering nested expressions in predicates of the WHERE clause. It is unclear whether the problem is in constructing the predicate representation or in rendering the representation (or both). Example: {code} SELECT o1.objectId FROM Object o1 WHERE ABS( (scisql_fluxToAbMag(o1.gFlux_PS)-scisql_fluxToAbMag(o1.rFlux_PS)) - (scisql_fluxToAbMag(o1.gFlux_PS)-scisql_fluxToAbMag(o1.rFlux_PS)) ) < 1 {code} Yields: {code} SELECT o1.objectId FROM LSST.Object_100 AS o1 WHERE ABS((VALUE_EXP FACTOR FUNCTION_SPEC scisql_fluxToAbMag(VALUE_EXP FACTOR COLUMN_REF o1.gFlux_PS)-FACTOR FUNCTION_SPEC scisql_fluxToAbMag(VALUE_EXP FACTOR COLUMN_REF o1.rFlux_PS)))<1 {code} I probably left out a more general implementation one/both of those parts of query parsing/analysis. | 4 |
| 888 | DM-1785 | Add rotAngle to baseline schema | Add "rotAngle DOUBLE" to every table that has image ra/decl. | 1 |
| 889 | DM-1786 | Implement using multiple disk spindles | Qserv should be able to take advantage of multiple disk spindles (JBOD-type architecture). In practice that means either relying on something like native mysql partitioning, or tweaking loader so that it can distribute chunks across multiple disks (and patch symlinks in mysql data_dir). | 20 |
| 890 | DM-1787 | OpenStack automation via Python scripts : Launch an Instance | In our introductory work with OpenStack we have been utilizing the Horizon GUI interface for first steps, followed by the use of command line tools (the 'CLI') (e.g., nova, cinder, etc) as shown in DM-1334 DM-1700 , DM-1701. While it is possible to write automation scripts that utilize the CLI, an approach based on 'pure' Python scripting would fit more seamlessly into the LSST software development process. Enabling OpenStack automation via Python offers the opportunity to integrate provisioning of resources into the overall flow of LSST workflow & processing (e.g., DRP.) The OpenStack services expose native Python APIs that expose the same feature set as the command-line tools. The required python packages (python-keystoneclient, python-novaclient, ..) are installed on the head node 'vlad-mgmt' of the NCSA ISL OpenStack, and so initial Python scripting can be executed/tested there. A first Python script will perform required authentication and launch an instance. | 4 |
| 891 | DM-1788 | OpenStack automation via Python scripts : Software installation/test on an LSST node | The set of packages that will enable us to write against the native Python APIs of the OpenStack services is {code} python-keystoneclient python-glanceclient python-novaclient python-quantumclient python-cinderclient python-swiftclient {code} We begin testing these in DM-1787 on the NCSA OpenStack head node, but eventual use within LSST orchestrating workflow would entail these being installed on LSST nodes in the LSST stack. In this issue we perform a basic installation of these packages into the system space on an LSST node/VM for testing. These are managed in github, and we install these via 'pip install' onto an LSST VM for initial tests. | 4 |
| 892 | DM-1789 | Experiment with afwtable, meas_base, pipe_{tasks, base} | NULL | 4 |
| 893 | DM-1790 | Produce detailed prototype & accompanying documentation | NULL | 4 |
| 894 | DM-1791 | Produce straw-man prototype | NULL | 3 |
| 895 | DM-1792 | Update documentation and automatic install script w.r.t. new newinstall.sh script | newinstall.sh script has evolved and breaks Qserv install procedure. | 1 |
| 896 | DM-1794 | Pull distEst package into obs_subaru | Reducing HSC data requires an estimate of the distortion, which is provided by the HSC package distEst. This can be pulled into obs_subaru to consolidate code and reduce dependencies. I propose to treat distEst as legacy code, which means I will pull it into obs_subaru without major changes to the code style. | 6 |
| 897 | DM-1795 | Use anonymous NCSA rsync server to distribute large test datafiles. | DM-1755 has been done before this feature was available. It uses rsync over ssh which require use to have an ssh-key on lsst-dev. NSCA rsync server can now be accessed with next syntax: {code:bash} rsync -av lsst-rsync.ncsa.illinois.edu::qserv/qserv_testdata/datasets/case04/data/DeepSource.csv.gz . # LIST FILES AVAILABLE IN THE MODULE NAMED "qserv" rsync lsst-rsync.ncsa.illinois.edu::qserv # To add content to this module/group, you can copy files into the following path: /lsst/rsync/qserv/ {code} rsync over ssh feature should be kept, in order to distribute private data are distributed. | 4 |
| 898 | DM-1796 | Improve test coverage for case04 | Most of queries used in GB-sized case04 return empty results. These queries should be added: {code:bash} fjammes@lsst-db2:~/src/qserv_testdata (u/fjammes/DM-1755) $ mysql --host 127.0.0.1 --port 4040 --user qsmaster qservTest_caseSUI_qserv -e "SELECT count(*) FROM DeepForcedSource" +----------------+ | SUM(QS1_COUNT) | +----------------+ | 33349940 | +----------------+ fjammes@lsst-db2:~/src/qserv_testdata (u/fjammes/DM-1755) $ mysql --host 127.0.0.1 --port 4040 --user qsmaster qservTest_caseSUI_qserv -e "SELECT * FROM DeepForcedSource LIMIT 1" {code} But, even better, SUI team could provide some more interesting query to Qserv team in order to improve case04 quality. | 3 |
| 899 | DM-1797 | Package flask | The Data Access Webservice APIs are relying on flask, so we need to package flask according to the LSST standards. For my initial testing, I just run "sudo aptitude install python-flask". | 1 |
| 900 | DM-1798 | Regression testing of AP Simulator | Run the AP simulator to make sure that none of the changes to the ctrl_events package break anything. | 1 |
| 901 | DM-1799 | Regression testing of Orca | Do some test runs using Orca to make sure Orca still works after the the changes to ctrl_events. | 1 |
| 902 | DM-1802 | remove unused local typedefs | gcc 4.8 now warns about locally-defined typedefs that aren't used. We have a few of these in ndarray and afw::gpu that should be removed. | 1 |
| 903 | DM-1803 | S15 Explore Qserv Authorization | Explore authorization centrally: use information generated by parser. Either generate dummy query and run on mysql that runs near czar, or use info produced by parser to determine if user is authorized. Note, we want to limit this to ~1 week, just to reveal potential problems, or do a quick proof of concept. | 8 |
| 904 | DM-1804 | Study the current SUI requirement | Study the current requirement carefully to make sure they all make sense and we can do it. | 4 |
| 905 | DM-1805 | Study the current SUI requirement | NULL | 4 |
| 906 | DM-1806 | Study the current SUI requirement | NULL | 2 |
| 907 | DM-1807 | Study the LSST data products document and give a summary to the team | NULL | 6 |
| 908 | DM-1808 | Study the current SUI requirement | NULL | 2 |
| 909 | DM-1809 | Study SUI requirement and summarize all the input from other team memebers | NULL | 10 |
| 910 | DM-1810 | segfaults in ip_diffim on gcc 4.8 | I'm seeing test segfaults in ip_diffim on gcc 4.8, similar to those resolved on DM-1725, but with no similar smoking gun yet. Preliminary indication is that the problem is actually in meas_algorithms. | 2 |
| 911 | DM-1811 | Prepare initial content | Prepare LSE-130 content as far as possible without input from the new collimated-projector calibration plan. | 4 |
| 912 | DM-1812 | Determine LSE-130 impact of collimated projector calibration plan | During a working meeting with Robert Lupton and Chris Stubbs, determine the impact on LSE-130 of the introduction of the collimated projector for calibration. | 8 |
| 913 | DM-1813 | Prepare draft of LSE-130 for Camera and CCB review | Produce a reviewable draft of LSE-130 based on decisions on calibration operations | 4 |
| 914 | DM-1814 | Support Camera CD-2 (mainly re: LSE-130) | Provide slides and other information needed for CD-2, mainly relative to the open questions around LSE-130 | 2 |
| 915 | DM-1815 | Support LSE-130 review by CCB (mainly Camera) | Respond to comments, perform revisions to LSE-130 as necessary based on feedback from CCB review of the document | 4 |
| 916 | DM-1816 | Convert LSE-130 to SysML | Following CCB recommendation of approval of LSE-130 draft, convert Word draft to SysML and provide a docgen to Robert McKercher for final posting. | 2 |
| 917 | DM-1817 | Create and post docgen of LSE-68 | To support discussions with the Camera, post a provisional docgen of LSE-68 to the appropriate Confluence page. Use knowledge from EA training to improve template. | 4 |
| 918 | DM-1818 | Support completion of final document | Based on CCB approval of LSE-72 on 10 October, support the completion of the final copy of the document for posting on Docushare. | 1 |
| 919 | DM-1819 | Complete LSE-140 work as needed to produce final document | Complete any review-driven revisions of LSE-140 and support the CCB meeting and following final document preparation. | 2 |
| 920 | DM-1820 | LSE-140: Collect desired changes for future release | Prepare for a future revision (Phase 3) of LSE-140. Collect issues to be addressed in the revision. Determine if any affect Phase 2 scope (which would require a prompt revision). It is not anticipated that there will be an actual revision of LSE-140 during the Winter 2015 cycle, because additional detail on calibration requirements will not be available in time. | 1 |
| 921 | DM-1822 | Fix czar assertion failure | Reported by Tatiana: I am encountering this once in a while. qserv-czar.log python: build/rproc/ProtoRowBuffer.cc:69: int lsst::qserv::rproc::escapeString(Iter, CIter, CIter) [with Iter = __gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, CIter = __gnu_cxx::__normal_iterator<const char*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > >]: Assertion `srcBegin != srcEnd' failed. Czar is dead and qserv stops responding after that. ---- For more details, search in qserv-l archives mails with "czar assertion failure" subject: https://listserv.slac.stanford.edu/cgi-bin/wa for complete description. | 4 |
| 922 | DM-1824 | Define issues to be addressed | Work with TCS contacts (Jacques Sebag, Paul Lotz, etc.) to define the principal issues | 1 |
| 923 | DM-1825 | Produce draft of LSE-75 with agreed revisions | Produce a draft of LSE-75 with the following agreed revisions: * remove reference to advance notice of pointing, now in LSE-72 * add reference to PSF reporting | 1 |
| 924 | DM-1826 | Catch-all epic for essential fixes during DM-W15-1,2,3 | NULL | 10 |
| 925 | DM-1828 | Create primary calibration plugins | NULL | 9 |
| 926 | DM-1829 | Develop General Acceptable Use Policy | Don Petravick and Lee LeClair | 6 |
| 927 | DM-1830 | Develop Information Classification Policy | Don Petravick, Lee LeClair | 6 |
| 928 | DM-1831 | Develop Incident Response Policy | Don Petravick, Lee LeClair | 6 |
| 929 | DM-1832 | Develop Data Management Sub-Project Plan and Risk Table | Don Petravick, Frossie Economou | 6 |
| 930 | DM-1833 | Develop PMO Sub-Project Plan and Risk Table | Lee LeClair, Iain Goodenow | 6 |
| 931 | DM-1834 | Develop EPO Sub-Project Plan and Risk Table | Don Petravick, Frossie Economou | 6 |
| 932 | DM-1835 | Develop Camera Sub-Project Plan and Risk Table | Don Petravick, Richard Dubois | 6 |
| 933 | DM-1836 | Develop Telescope and Site Sub-Project Plan and Risk Table | Lee LeClair, German Schumacher | 6 |
| 934 | DM-1837 | Package ISP Documents into LSST Standard Format for Control and Delivery | Robert McKercher | 4 |
| 935 | DM-1839 | Deblend post-merge objects | NULL | 4 |
| 936 | DM-1840 | Task-level processing for merged objects | NULL | 5 |
| 937 | DM-1841 | Fix query error on case03: "SELECT scienceCcdExposureId FROM Science_Ccd_Exposure_Metadata" | Xrootd prevents the worker to return more than 2MB data. On GB-sized data: {code} mysql --host=127.0.0.1 --port=4040 --user=qsmaster --batch -e "SELECT scienceCcdExposureId FROM Science_Ccd_Exposure_Metadata" ERROR 4120 (Proxy) at line 1: Error during execution: -1 Ref=1 Resource(/chk/qservTest_case03_qserv/1234567890): 20150123-16:27:45, Error merging result, 1420, Result message MD5 mismatch (-1) {code} On integration test case 04: {code} qserv@clrinfoport09:~/src/qserv (u/fjammes/DM-1841 *)⟫ mysql --host=127.0.0.1 --port=4040 --user=qsmaster qservTest_case04_qserv -e "SELECT * FROM DeepForcedSource" ERROR 4120 (Proxy) at line 1: Error during execution: -1 Ref=1 Resource(/chk/qservTest_case04_qserv/6970): 20150204-16:23:43, Error merging result, 1420, Result message MD5 mismatch Ref=2 Resource(/chk/qservTest_case04_qserv/7138): 20150204-16:23:43, Error merging result, 1420, Result message MD5 mismatch Ref=3 (-1) {code} | 5 |
| 938 | DM-1843 | Permit PropertySets to be represented in event payloads | In the old marshalling code, property sets were representable within the payload of the event. This was removed in the new marshalling scheme. There are things (ctrl_orca) that still used this, so this needs to be added to the new marshaling code. At the same time, new new filtering code can not allow this to be added, because the JMS headers only take simple data types. | 2 |
| 939 | DM-1844 | Test Qserv on SL7 | Needed to run Qserv on CC-IN2P3 cluster. | 2 |
| 940 | DM-1845 | Coordinate implementation of web form for collecting data about existing data sets | The form is being implemented by the DataCat team. Need to coordinate (including with the NCSA team which parts are covered by which team), test, fine tune etc. | 4 |
| 941 | DM-1847 | Add support for large results in XrdSsiRequest::GetResponseData | GetResponseData needs to handle data sets beyond 2 MB. The problem is discussed in more details in story DM-1841 | 12 |
| 942 | DM-1848 | SUI work with DB team to define the image query API | IPAC SUI team will work with SLAC database team to define the image query APIs. IPAC needs to make sure the APIs are sufficient to satisfy the UI needs. SLAC will implement them. . | 10 |
| 943 | DM-1849 | Study RESTful API and work with SLAC team to define image query APIs | NULL | 2 |
| 944 | DM-1850 | Discuss, review, and define image query APIs with SLAC team | NULL | 2 |
| 945 | DM-1851 | Image query API discussion and review | NULL | 2 |
| 946 | DM-1852 | Image query API discussion | NULL | 1 |
| 947 | DM-1853 | Image query API discussion | NULL | 1 |
| 948 | DM-1854 | SUI propose a structure definition for user workspace | Workspace is an integral part of SUI. We want to start the discussion and definition of workspace concept and structure. SUI team had several discussions and Xiuqin presented the results at the DM AHM at SLAC. The slides and the discussion notes are here: https://confluence.lsstcorp.org/display/DM/Workspace+discussion | 20 |
| 949 | DM-1855 | Specify mechanism for periodic (nightly/weekly) build distribution | NULL | 3 |
| 950 | DM-1856 | Implement nightly/weekly release automatic distribution - Part I | This ticket covers code in sqre-codekit to do migrate as much of the process of special machines, and git tag repos on the basis of eupspkg manifests. | 4 |
| 951 | DM-1857 | Github Transition Plan: Write document for CCB | NULL | 7 |
| 952 | DM-1858 | Release engineering W15 bucket | Bucket epic for activities related to stack releases during W15 | 24 |
| 953 | DM-1859 | Publish v10_0 release | NULL | 12 |
| 954 | DM-1860 | Update documentation for v10_0 release | All done bar obtaining some release notes. | 2 |
| 955 | DM-1861 | Workflow tool improvements w15 bucket | Bucket epic for w15 for improvements with JIRA, Hipchat, etc | 12 |
| 956 | DM-1862 | JIRA project for RFCs | NULL | 3 |
| 957 | DM-1864 | Implement Github transition plan | NULL | 10 |
| 958 | DM-1865 | Review existing Level 3 documentation | Review existing requirements in this area. Find all relevant existing project-controlled and other key documents. | 4 |
| 959 | DM-1866 | Document as-is Level 3 requirements and conceptual design | Produce a single jumping-off point for documentation on all aspects of Level 3, on Confluence. Ensure that flowdown for existing Level 3 requirements in SysML is modeled. Describe the high-level conceptual design. | 4 |
| 960 | DM-1867 | tighten control over heterogeneous DictFields | DM-1218 added support for DictFields with heterogeneous item types, which probably allows a bit too much freedom (the rest of pex_config is much more strongly-typed). Instead of passing None to allow any type to be used, we should pass a tuple of supported types. | 1 |
| 961 | DM-1868 | Define JSON Results for Data Access Services | As discussed at [Data Access Hangout 2015-02-23|https://confluence.lsstcorp.org/display/DM/Data+Access+Hangout+2015-02-23], we should support json format. This story covers defining structure of JSON results for Data Access Services (dbserv, imgserv, metaserv) | 3 |
| 962 | DM-1871 | SUI requirement refinement to define many unclear areas | In the current SUI requirement document, many areas are not clearly defined. We want to put more description and definition for those areas, and identify and define the missing functions. The goal is to generate a requirement document for DM review and put it under version control. | 50 |
| 963 | DM-1872 | SUI User workspace specification | At the 2015 Deb DM AHM at SLAC, SUI led a discussion of workspace. https://confluence.lsstcorp.org/display/DM/Workspace+discussion We want to continue the discussion, understand the user needs, identify the DM groups involved. The goal is to generate a document to capture the functions requirement of workspace. The first version of document is here https://confluence.lsstcorp.org/pages/viewpage.action?pageId=41783931 | 30 |
| 964 | DM-1873 | SUI 2D data visualization (XY plot) | Better algorithm in spatial binning to visualize large number of catalog sources Plot histogram for tabular data Plot basic light curve | 40 |
| 965 | DM-1874 | SUI the alert subscription system specification | Identify parties involved in the alert system generation, broadcast, subscription. Understand the flow of the alert from generation to notifying users. Understand the requirement for SUI subsystem "Alert subscription and notification". | 30 |
| 966 | DM-1875 | SUI infrastructure implementation | Identify the hardware resources needed at NCSA for short term development and Set up the basic git repository and build system Explore multi resolution images display for background iamge | 40 |
| 967 | DM-1876 | SUI functional design | understand current use cases, collect and define more use cases. Design the major functions of major components in SUI, mainly Firefly package. | 60 |
| 968 | DM-1877 | SUI web interface and Python interaction | To facilitate users to interact with Firefly visualization components in IPython notebook, to allow users to control the display with Python script. | 60 |
| 969 | DM-1878 | Collect, understand, and define more use cases | This is an on-going effort. The collected use cases will be posted at confluence page https://confluence.lsstcorp.org/pages/viewpage.action?pageId=41784036. | 20 |
| 970 | DM-1880 | Implement RESTful interfaces for Database (GET) | Implement RESTful interfaces for Database (see all D* in https://confluence.lsstcorp.org/display/DM/API), based on the first prototype developed through DM-1695. The work includes adding support for returning appropriately formatted results (support the most common formats). This covers "GET" type requests only, "POST" will be handled separately. | 5 |
| 971 | DM-1881 | Improvements to web form | NULL | 5 |
| 972 | DM-1883 | organize the workspace discussion and present a good proposal | SUI team had several discussions and Xiuqin presented the results at the DM AHM at SLAC. The slides and the discussion notes are here: https://confluence.lsstcorp.org/display/DM/Workspace+discussion | 10 |
| 973 | DM-1884 | summarize WEBDAV capabilities and past experience using it | WEBDAV could be a candidate for managing the user workspace. summarize its capabilities and past experience, collect some use cases will help us to make a better decision. | 4 |
| 974 | DM-1885 | Contribute to the workspace capability discussion | This include past experience, collection of use cases. | 2 |
| 975 | DM-1887 | HDF5 file format study | Xiquin, Loi, Trey, and myself discussed HDF5 as a default format to return result set and metadata from lower-level database services vs. traditional IPAC table. Here is the summary: Advantages of IPAC Table format - Simple and human-readable, contains a single table - Fixed length rows (easy to page through) - Supported by many astronomical tools - Provides a way to pass data type, units, and null values in the header - More metadata can be added through keywords (attributes) Disadvantages of IPAC table format - Steaming can not be started before all data are received – need to know column width before the table can be written (csv is better alternative) - Only alpha-numeric and '_' characters are allowed in column names (small subset of available characters) - Only predefined datatypes and one attribute type (string) - ASCII representation requires about twice as much storage to represent floating-point number data than the binary equivalent. Advantages of HDF5 - Can represent complex data and metadata (according to LOFAR, good to represent time series) - Structured data, arbitrary attribute types, datatypes can be combined to create structured datatypes - Flexible datatypes: can be enumerations, bit strings, pointers, composite datatypes, custom atomic datatypes - Access time and storage space optimizations - Partial I/O: “Chunked” data for faster access - Supports parallel I/O (reading and writing) - Built-in compression (GNU zlib, but can be replaced with others) - Existing inspection and visualization tools (HDFView, MATLAB, etc.) Disadvantages of HDF5 - Complex - Tuned to do efficient I/O and storage for "big" data (hundreds of megabytes and more), not efficient for small reads/writes. - Requires native libraries (available in prepackaged jars, see below) - Not human readable - (?) Not yet widely supported by astronomical tools (counter-examples: AstroPy, IDL, more at hdfgroup site) Tools and Java wrappers: * JHI5 - the low level JNI wrappers: very flexible, but also quite tedious to use. * Java HDF object package - a high-level interface based on JHI5. * HDFView - a Java-based viewer application based on the Java HDF object package. * JHDF5 - a high-level interface building on the JHI5 layer which provides most of the functionality of HDF5 to Java. The API has a shallow learning curve and hides most of the house-keeping work from the developer. You can run the Java HDF object package (and HDFView) on the JHI5 interface that is part of JHDF5, so the two APIs can co-exist within one Java program. (from StackOverflow answer, 2012) * NetCDF-Java is a Pure Java Library, that reads HDF5. However, it's hard to keep pure java version up-to-date with the standard, does not support all the features. A way to set up native libraries (3rd option from JHDF5 FAQ): "Use a library packaged in a jar file and provided as a resource (by putting the jar file on the class path). Internally this uses the same directory structure as method 2., but packaged in a jar file so you don't have to care about it. Jar files with the appropriate structure are cisd-jhdf5-batteries_included.jar and lib/nativejar/.jar (one file for each platform). This is the simplest way to use the library." | 1 |
| 976 | DM-1889 | S15 Butler (v3) | Improvements and tweaks to the butler as needed. | 19 |
| 977 | DM-1890 | Research support for integrated view of all databases | Metadata Store needs to contain information about all databases (data release databases, Level 1, Level 3 user databases). | 11 |
| 978 | DM-1891 | Add support for IPAC table format | Implement support for result formatting in IPAC table format. | 6 |
| 979 | DM-1893 | Research supporting cutout from images with overlaps | Research producing cutout from images with overlaps. | 10 |
| 980 | DM-1894 | SUI refactor Firefly package | The team has studied and researched the web UI framework (DM-1148) in W15. A new framework will be decided in February 2015. Many classes in Firefly package need to be refactored to use the the new framework. This epic in S15 will be our first attempt for it. Developers will be involved in this effort: Trey Roby, Loi Ly, Tatiana Goldina, Lijun Zhang, Xiuqin Wu | 100 |
| 981 | DM-1896 | Design CSS schema to support table deletion | Table/chunk deletion can be an extended process as some worker nodes may be temporarily down. We need to define a process and its supporting structures in CSS to allow gradual deletion of individual chunks and full tables. Deliverable: a design of a system capable of deleting a distributed table (all chunks, all replicas). It should be possible to create a table with the same name after deletion. | 4 |
| 982 | DM-1897 | Modify CSS structure to support table deletion | Modify CSS structures to support DROP TABLE, as defined in DM-1896. | 2 |
| 983 | DM-1898 | Consistency checking for table data CSS | CSS data on tables/chunks/nodes is supposed to be consistent at all times. Would be nice to have a tool that verifies consistency, probably including checking actual worker state. | 4 |
| 984 | DM-1899 | Tool to dump CSS information | CSS information tree may become large and it would be nice to have a tool to examine that tree or parts of it. Something that dumps the tree in user-friendly way and allows filtering or summarizing. | 2 |
| 985 | DM-1900 | Worker management service - design | We need to replace direct worker-mysql communication and other administrative channels with a special service which will control all worker communication. Some light-weight service running alongside other worker servers, probably HTTP-based. Data loading, start/stop should be handled by this service. | 5 |
| 986 | DM-1901 | Re-implement data loading scripts based on new worker control service | Once we have new service that controls worker communication we'll need to reimplement WorkerAdmin class based on that. | 8 |
| 987 | DM-1903 | Implementation of calibration transformation framework | Following DM-1598 there will be a detailed design and prototype implementation for the calibration & ingest system. This issue covers cleaning up that code, documenting it, having it reviewed, and merging to master. | 2 |
| 988 | DM-1904 | Continued footprint improvements | A redesigned API and support for topological operations within the Footprint class. This continues the work started in DM-1107 in W15. Breakdown: jbosch 15%; swinbank 85% | 8 |
| 989 | DM-1905 | QSERV issues when working with scisql_s2PtInCircle | This problem has been adressed un u/fjammes/DM-1841. Here's Tatiana report: {quote} This error happens on all DeepForcedSource queries. (It happens on DeepSource too, but not always.) [2015-01-15 11:02:29] [Proxy][4120] Error during execution: -1 Ref=1 Resource(/chk/LSST/6970): 20150115-11:01:05, Complete (success), 0, Ref=2 Resource(/chk/LSST/7138): 20150115-11:02:19, Complete (success), 0, Ref=3 Resource(/chk/LSST/7140): 20150115-11:02:18, Error merging result, 0, Ref=4 Resource(/chk/LSST/730 (-1) Query examples: select * from DeepForcedSource where ra>0.4 and ra<0.6 and decl>0.9 and decl<1.1; select * from DeepForcedSource where scisql_s2PtInCircle(ra, decl, 0.5, 1.1, 0.138) = 1; {quote} | 2 |
| 990 | DM-1906 | Outline an expandable Python framework for advanced users | Outline an expandable Python framework for use by advanced users in interactive or batch mode. Many users should be able to contribute software to the framework following simple API guidelines. Also look at configuration and delivery systems for the software. Start by looking for existing Python software that could be used in the areas of: - Database & file access - Data analysis frameworks - Display of data,, especially in an Astronomical context. - Reading and writing data in different formats - Graphing data locally or over the web - Science and astronomy data analysis - Event interfaces - VO functionality and connecting with existing software like DS9,, Aladin, Topcat, etc. - Ways to integrate modules with an LSST focus like AGN Large Scale Structures Galaxies Local Volume Solar System Astrostatistics Stellar Pops Strong Lensing Supernova Transients Weak Lensing Camera - Configuration and delivery system | 12 |
| 991 | DM-1907 | Backport HSC multi-band deblend processing | Breakdown: lauren 60%; price 40% We need to transfer the recent HSC-side multi-band deblender changes to the LSST side, including all HSC issues in this query: https://hsc-jira.astro.princeton.edu/jira/issues/?filter=11603 | 35 |
| 992 | DM-1909 | Interface design for full focal plane PSF estimation | Breakdown: swinbank 40%; jbosch 30%; rhl 30% This is mostly design work, but I'd like the goal to be a new Python command-line task that repeats as much of the current ProcessCcd as necessary to run full focal plane PSF estimation, and would serve as a starting point for a future-proof visit processing script. Some issues include: - Gather requirements from e.g. DESC people as to needed inputs and data flow. - Get details of what camera/telescope systems will provide, and figure out how those related to what DESC needs. - Design classes for camera/telescope engineering data and wavefront information. Figure out how they'll be managed on disk (stored with Exposure, part of CameraGeom, with flats, biases, etc.). - Discuss parallelization needs with middleware team, and determine a way forward that lets us design interfaces using future parallelization schemes that don't exist yet. - Design Python task interface for full focal plane PSF estimation. - Implement command-line task that calls the full focal plane PSF estimation task. - Implement placeholder PSF estimation subtask that just uses existing PSF determiners on single CCDs. This may be too ambitious for the 40 story points we've allocated, and should discuss either adding more effort or reducing the scope. | 45 |
| 993 | DM-1910 | DRP DM-S15-1,2,3 Bugs and Papercuts | Breakdown: jbosch 16%; lauren 16%; rhl 20%; pgee 16%; price 16%; swinbank 16% | 20 |
| 994 | DM-1912 | DRP DM-S15-4,5,6 Bugs and Papercuts | Breakdown: jbosch 16%; lauren 16%; rhl 20%; pgee 16%; price 16%; swinbank 16% | 20 |
| 995 | DM-1913 | Prototype command interaction with Firefly | NULL | 10 |
| 996 | DM-1914 | Research Javascript Frameworks: Finish new framework proposal | NULL | 18 |
| 997 | DM-1915 | Create a deployable installation package for Firefly | With Firefly being open-source, we should provide a simple all-in-one installation package so a user can quickly setup and deploy an instance the Firefly Tools web application. | 10 |
| 998 | DM-1916 | Fine-tune data access interfaces | Brought up by Gregory via comments on the API page. Data release selection in queries: I see that the /db/... queries take a "?" query parameter "db" with an example value of "DR1", i.e., a data release selector. A couple of remarks: * Will this query parameter be provided for all the Level 2 image data products, e.g., for retrievals of coadded images? **If so, then it needs an equivalent to the M4 "GET /meta/v0/db/<type>" query. * I assume the "db" query parameter defaults to the most recent data release. * Will the M4 query return an indication of which "?db=" value is the current default? * I assume that the numeric-identifier components of the various paths are unique only within a single data release. That means that eventually, in user documentation, we should make sure that they understand that they can't scan through different releases' versions of the same image (for example) just by varying the "?db=" parameter. * Are the identifiers also unique within a particular type (i.e., "raw", "template", "coadd", "calexp", etc.)? ----- Distinguishing L1 and L2 versions of reprocessed data products Since most or all of the L1 data products will be regenerated in each data release, the catalog and image APIs should presumably allow the user to distinguish between the two. I see how this could be done for catalogs - the "?db=" parameter presumably allows selecting something like "L1" (for the actively updated Level 1 database) in addition to the above-documented "DR1", "DR2", etc. Will the L2 table names for the reprocessed L1 data products be generally expected to be the same as for L1? (Barring the discovery of a serious issue that requires revision of the schema for the reprocessing.) How will the L1 and reprocessed-L1 image data products be distinguished? | 9 |
| 999 | DM-1917 | Fix missing virtual destructors | The compiler is warning about some derived class hierarchies that are lacking virtual destructors. We should add at least empty implementations to the base classes of these hierarchies. | 1 |
| 1000 | DM-1918 | write Unit test and validation classes to validate the FITSreader refactoring | NULL | 8 |
| 1001 | DM-1919 | Address misc. compiler warnings | Fix places where compiler is warning about some things we are doing on purpose and which we don't intend to change. This helps keep compiler noise down so its easier to notice "real" warnings. | 1 |
| 1002 | DM-1920 | update shapeHSM wrappers to latest external version | The HSM shape code has undergone many improvements and bug fixes as part of being included in the GalSim package, and we've recently included those in the HSC fork of meas_extensions_shapeHSM (HSC-129, HSC-1093). We should transfer those changes to the LSST side before tackling DM-981 (or at least before finishing it). The story point estimate here is for the work already done on the HSC side (with the usual 50% factor for shared work). The transfer to the LSST side should be essentially no effort. To the extent that EVM cares about this, the credit should go to [~price], even though I ([~jbosch]) am doing the transfer. | 6 |
| 1003 | DM-1921 | Make unit tests use shared libraries | Many (all?) unit tests are currently built as static executables which include all needed object files. This has several issues associated with it: - many files are compiled twice, once as *.os files for shared libraries, second time as *.o file for unit tests - unit tests do not test actual code in the shared libraries but instead separately-built copy of the same code We should change our procedure and make unit test to link against shared libraries to avoid these problems. | 4 |
| 1004 | DM-1923 | Setup network for IPMI | NULL | 2 |
| 1005 | DM-1924 | Setup IPMI bastion hosts | NULL | 5 |
| 1006 | DM-1925 | Document how to use IPMI with LSST infrastructure | NULL | 1 |
| 1007 | DM-1926 | Base configuration of NFS servers | install and configure OS | 3 |
| 1008 | DM-1927 | Test new NFS servers | Test to confirm servers are configured optimally | 15 |
| 1009 | DM-1928 | Deploy first of the NFS servers | NULL | 20 |
| 1010 | DM-1929 | LOE - Week ending 12/26/14 | NULL | 4 |
| 1011 | DM-1930 | LOE - Week ending 1/9/15 | NULL | 8 |
| 1012 | DM-1931 | LOE - Week ending 1/16/15 | NULL | 4 |
| 1013 | DM-1932 | LOE - Week ending 1/23/15 | NULL | 8 |
| 1014 | DM-1934 | LOE - Week ending 2/6/15 | NULL | 8 |
| 1015 | DM-1935 | LOE - Week ending 2/13/15 | NULL | 8 |
| 1016 | DM-1936 | LOE - Week ending 2/20/15 | NULL | 8 |
| 1017 | DM-1937 | LOE - Week ending 2/27/15 | NULL | 8 |
| 1018 | DM-1939 | Define instrumental inputs to PSF estimation | Define inputs needed to build physically-motivated PSF models beyond what's contained in the image data and the current CameraGeom. This includes: - static engineering data from lab tests - slowly-varying engineering data (measured daily, weekly, etc.) - per-visit auxiliary data from telescope and camera (including, but not limited to wavefront sensor data). The focus here should be on APIs, not the technical details of the data; we want to define class hierarchies (perhaps some polymorphic ones) that can be used to pass this data around in the future. We also want to identify and characterize any preprocessing that needs to be done before they can be used for PSF estimation. Experts who should be consulted include [~gpdf] (who is in charge of making sure the camera and telescope interfaces to DM are well-defined), camera/telescope people he recommends, and people from the LSST DESC who have worked on physical PSF estimation and know something about the things they'll want (e.g. Michael Schneider, Aaron Roodman). This is essentially a requirements-gathering task, so the output should be a confluence page, not code. | 10 |
| 1019 | DM-1940 | API for instrumental inputs to PSF estimation | Following the requirements-gathering in DM-1939, come up with classes that can be used to pass the needed inputs to the PSF estimation code, and at least sketch out roughly how they will need to be managed by the butler and loaded by the framework code. The output of this ticket should be an API design in confluence (possibly on the same page used for DM-1939), and an associated RFC. | 10 |
| 1020 | DM-1941 | Organize SUI design discussions | Organize the SUI team for web UI discussions to capture as much functions and components as possible. A draft design document should be produced out of those discussions. | 10 |
| 1021 | DM-1942 | Test data development and HSC stack integration | Breakdown: price 20%; lauren 80% This epic is focused on general stack testing and integration on HSC data, with a goal of getting LSST-side reductions of HSC data to the same level of quality and robustness currently present on the HSC fork of the stack, which will involve a combination of backporting minor fixes from the HSC codebase and tuning parameters for LSST-side algorithms that supercede their HSC-side counterparts. This will allow science-level algorithms to be tested using HSC data, and will provide functionality important for science-grade tests on other real data (such as the ongoing CFHTLS reprocsessing at IN2P3). This effort will be focused on single-epoch processing (i.e. ProcessCcdTask), with coadd-level processing a stretch goal dependent somewhat on the completion of astrometric calibration work at UW. | 45 |
| 1022 | DM-1943 | HSC backport: convert Peak to PeakRecord | This issue covers transferring all changesets from [HSC-1074|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1074] and its subtasks, as well as: - An RFC to propose the API change, and any requested modifications generated by the RFC. - Additional fixes to downstream code that's broken by this change (HSC-side changesets should be present for most of downstream fixes, but perhaps not all). | 8 |
| 1023 | DM-1944 | HSC backport: guarantee consistent handling of peaks in deblender | This issue covers transferring changesets from: - [HSC-134|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-134] - [HSC-1109|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1109] - [HSC-1083|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1083] | 4 |
| 1024 | DM-1945 | HSC backport: multiband processing for coadds | This issue includes transferring changesets from many HSC issues: - HSC-1060 - HSC-1064 - HSC-1065 - HSC-1061 Most of this is in multiBand.py in pipe_tasks, but there are scattered changes elsewhere (including updates to camera mappers to include the new datasets, for which we'll need to modify more than just obs_subaru). However, before we make these changes, we'll need to open an RFC to gather comments on the design of this task. We should qualify there that this is not a long-term plan for consistent multiband processing (which we'll be starting to design on DM-1908), but a step towards better processing in the interim. Note: while I've assigned this to [~lauren], as I think it will be very helpful for her to get familiar with this code by doing the transfers, the RFC will have to involve a collaboration with [~jbosch], [~price], and Bob Armstrong, as we can't expect someone who wasn't involved in the design to be able to write a document justifying it. | 8 |
| 1025 | DM-1946 | HSC backport: low-level Footprint merge code | This is a transfer of changesets from the follow epics: - [HSC-1020|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1020] - [HSC-1075|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1075]: only the afw changes Because this is purely an addition to the interface, and we're planning to redesign that interface in DM-1904, I don't think we need an RFC here. | 4 |
| 1026 | DM-1947 | Add support for async request cancellation to xrdssi | NULL | 12 |
| 1027 | DM-1948 | S15 Implement Query Cancellation | Add support for query cancellation. | 32 |
| 1028 | DM-1950 | Add abstraction in czar for unit tests | Add hooks in czar that will let us build unit tests. | 8 |
| 1029 | DM-1951 | Unit test for query cancellation | NULL | 8 |
| 1030 | DM-1952 | Change log priority for message "Unknown column 'whatever' in 'field list'" | Next message should be logged with ERROR priority: {code} 0204 15:08:03.748 [0x7f1f4b4f4700] INFO Foreman (build/wdb/QueryAction.cc:250) - [1054] Unknown column 'whatever' in 'field list' {code} | 1 |
| 1031 | DM-1953 | Post meas_base move changes to Kron | These are to note leftovers from DM-982. They could be done in a single issue. 1. I commented code out referring to correctfluxes, but it will need to be restored once it is available in the new framework. 2. Jim asked me to replace the computeSincFlux which is currently in PsfImage.cc in meas_algorithms with a similar call in meas_base/ApertureFlux.cc. I did not do this because it became rather complicated, and can just as easily be done when the meas_algorithms routine is moved or removed. Basically, the templating in ApertureFlux is on Pixel type, whereas in meas_algorithms it is on ImageT (where ImageT is not necessarily a single class hierarchy -- e.g., Image and MaskedImage). So I left this for now. | 1 |
| 1032 | DM-1954 | HSC backport: deblended HeavyFootprints in forced photometry | This is a transfer for changesets for [HSC-1062|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1062]. Unlike most of the HSC backport issues for multiband deblending, these changes will require significant modification the LSST side, because we need to apply them to the new forced measurement framework in meas_base rather than the old, HSC-only one in meas_algorithms and pipe_tasks. Also include [HSC-1256|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1256], [HSC-1218|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1218], [HSC-1235|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1235], [HSC-1216|https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1216]. | 20 |
| 1033 | DM-1955 | International Network Design and Implementation | FIU/Amlight is expected to provide a full capacity 3 x100 Gbps link by FY17. | 12 |
| 1034 | DM-1956 | REUNA will provide a “pre-operations” link between La Serena and Santiago | REUNA will provide La Serena - Santiago links at full capacity 100 Gbps link by FY17 and a 40 Gbps secondary link by FY20. This link will support testing and development prior to that time. | 14 |
| 1035 | DM-1957 | Chilean National Network Design and Implementation | Refer to REUNA MREFC sub award contract for deliverable details. This covers non-contract work by the AURA/LSST. | 20 |
| 1036 | DM-1959 | S15 Data Distribution & Replica Mgmt Prototype | This epic covers building an initial prototype of the Data Distribution and Replication system. The design is covered through DM-1060 | 30 |
| 1037 | DM-1961 | Prepare data set for large scale tests | Load data set for large scale tests (duplicate, partition, load) | 10 |
| 1038 | DM-1962 | Run large scale tests | Coordinate running large scale tests | 6 |
| 1039 | DM-1964 | Parallelization requirements for PSF estimation | We need to gather algorithmic ideas for how full focal plane PSF estimation will work from a parallelization and data flow standpoint, and discuss with the middleware team how these should be handled from an interface standpoint. Questions include: - Will we need to do significant cross-CCD image processing or require significant memory for these tasks? If so, should we structure this via message passing between CCD-level processes, or scatter-gather? - Assuming a scatter-gather approach, will we need multiple scatter/gather iterations when processing a single visit? - How much data will be passed between threads/processes? Would this include complex serializable objects, or just POD arrays? - Will different PSF estimation plugins will have different parallelization requirements? Or, can we define the plugin interfaces at a low-enough level that parallelization can be handled by the framework? If plugins do need to control their own parallelization, how do we make parallelization interfaces accessible to the plugins? | 8 |
| 1040 | DM-1965 | Python interface for full-visit PSF estimation | Create a Python interface for a pluggable PSF estimation system that supports algorithms that will operate over full images. This should include a sketch of how a calling command-line task, and placeholders for parallelization interfaces that may not yet be finalized. The output of this issue is a completed RFC on the design. | 6 |
| 1041 | DM-1966 | Command-line driver and placeholder implementation for PSF estimation | Create a command-line task that makes use of the new PSF estimation interface, duplicating as much of ProcessCcdTask's functionality as necessary to provide the inputs to PSF estimation (I expect this new task to ultimately replace ProcessCcdTask). This may have to include workarounds or temporary implementations for parallelization features that are not yet available. Create a simple PSF estimation placeholder that simply uses existing single-CCD PSF-determiners. | 6 |
| 1042 | DM-1968 | CModel flux validation and testing | Investigate the performance of the new version of the CModel code on various test datasets, including HSC data (following DM-1942), SDSS, and possibly CFHT data. Breakdown: lauren 100% | 35 |
| 1043 | DM-1969 | Create a kind of Wcs that encapsulates a TAN WCS and a distortion model | We can simplify the astrometry solver if we have a Wcs that encapsulates a pure tangent-plane WCS and a distortion model that takes converts between PIXELS and TAN_PIXELS. This is useful because at the early stages of processing raw data we have a TAN WCS from the telescope control system and a pretty good estimate of distortion from the optical model (represented in the camera geometry). The result will be a Wcs whose sky<->pixel transformation is a reasonable approximation of reality (and a much better approximation than the tangent-plane WCS that is currently available. This will potentially eliminate a significant amount of confusing code that attempts to correct for distortion by manually applying the optical distortion model. | 4 |
| 1044 | DM-1973 | Build 2015_02 Qserv release | See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe. | 1 |
| 1045 | DM-1974 | Fix enclose, escape, and line termination characters in qserv-data-loader | Add this string to mysql loader 'LOAD DATA INFILE' command: {code} q += "ENCLOSED BY '%s' ESCAPED BY '%s' LINES TERMINATED BY '%s'" % (enclose, escape, newline) {code} and add params in cfg file. | 2 |
| 1046 | DM-1977 | S15 Implement Fully-RESTful Data Access Web Service | Improvements to the skeleton of Data Access Web Service built in W15. Make all responses fully RESTful (including results, and errors, setting response headers). Fine-tune API, add support for versioning, unit testing. | 39 |
| 1047 | DM-1978 | Research and Document API Versioning | Research and document versioning of the RESTful API (through flask blueprints). In particular, need to understand how to avoid code duplication between different versions of API. | 4 |
| 1048 | DM-1979 | Add SQLite-based v0.1 unit testing for metaserv | Add unit tests for the RESTful flask based API. I think it'd be most useful if we could a) load some test data into underlying metaserv b) run programmatically things like "curl -H accept:text/html http://127.0.0.1:5000/meta/v0/db/L2/DC_W13_Stripe82/tables" etc (more examples in dax_*serv/README.txt) and verify that we got what we expected. Do it for both json and html. | 4 |
| 1049 | DM-1980 | Add error handling for webserv | Add basic error handling for the RESTful flask-based API | 6 |
| 1050 | DM-1981 | Improve security for mysql in python | Revisit all code that talks to mysql from python to use parameter bindings instead of direct string substitutions. In practice: {code} conn.execute("SELECT * FROM t WHERE name=%s" % theName) {code} should be replaced with {code} conn.execute("SELECT * FROM t WHERE name=%s", (theName,)) {code} For details, see http://mysql-python.sourceforge.net/MySQLdb.html#some-examples | 5 |
| 1051 | DM-1982 | Fix JDBC timestamp error | JDBC driver returns an error on next query: {code:sql} sql> select * from Science_Ccd_Exposure [2015-02-06 13:39:37] 1 row(s) retrieved starting from 0 in 927/970 ms [2015-02-06 13:39:37] [S1009] Cannot convert value '0000-00-00 00:00:00' from column 32 to TIMESTAMP. [2015-02-06 13:39:37] [S1009] Value '[B@548997d1' can not be represented as java.sql.Timestamp {code} | 1 |
| 1052 | DM-1987 | Redesign/Refactor WCS and Coord | %50 KSK, %50 RO Currently WCS is mutable and Coord objects are heavyweight. Refactor WCS to be immutable and make Coord less heavyweight. Include lists of Coord objects. It's possible astropy could inform in that area. Also, remove TanWcs in favor of TanSipWcs since TanWcs can have SIP terms. | 40 |
| 1053 | DM-1988 | Update analysis tasks: diffim and snap combination | 50% KSK 50% RO A small amount of this work is in 02C.03.01 The diffim task needs to be looked at. It needs to be updated to use current mechanisms. It should also be refactored to split out some of the procedural code into methods. In a similar task, the lsstSimIsrTask needs to include a real snap combine step. Currently, one of the snaps is dropped on the floor. For this round just implement naive snap addition and morphological CR rejection. | 43 |
| 1054 | DM-1989 | Define API for Stack Astrometric Calibration | 70% RO 30% KSK This should also include a minimal implementation. This should be done with an eye toward photometric calibration. Prerequisite: Get multi epoch (multi-band?) catalogs of centroids from some trusted source (CFHT, HSC?). 1. Load all stars that overlap a patch for all epochs. 2. Associate all stars on each chip. 2a. Implement K-D tree 3. Fit model for rigid chip system + optics + atmosphere. Eigen for sparse model fitter. 3a. Allow for external catalog. 3b. This could include a class to fit XYTransforms 4. Turn result into WCS. So maybe a down scope for a single cycle epic is to get matches and interfaces for models and solvers. | 75 |
| 1055 | DM-1990 | Research DCR in the context of DiffIm including possible algorithms for mitigation. | It is not clear how template coadds will be built. This includes understanding the data necessary to generate a template for the entire sky. This epic is to identify possible techniques as well as the risks associated with each technique. This does not need to pin down the exact algorithm or the specific selections, but should inform what further development is necessary to avoid putting alert generation at risk. | 19 |
| 1056 | DM-1991 | Refactor Approximate and Interpolate classes | 100% RO A base class for this will be created in DM-740 as a part of epic DM-85. This epic is to implement the classes to replace the original functionality. | 34 |
| 1057 | DM-1992 | SQuaRE Support | 50% KSK 50% RO SQuaRE has asked that we leave 20 SP free per Cycle to help out. | 20 |
| 1058 | DM-1994 | Story point display and roll-up in epic display | I understand that there is a pending request to display the story points for individual story issues in the mini-table in which they are displayed for an epic. It would also be useful to see a rolled-up total of the story points for the defined set of stories - so that, among other things, this could be compared to the story point value for the epic. Ideally the story points for the roll-up might be displayed as "nn (mm)" where nn is the total points and mm is the number of points remaining to do (or done already - I don't care which as long as the definition is clear). | 1 |
| 1059 | DM-1996 | Define faulty/consistent states and recovery process | Whiteboard session(s) to gather a list of invariants / principles that define a consistent state, a list of fault conditions, and the steps in failure recovery. | 12 |
| 1060 | DM-1997 | Data transport mechanism for data distribution | Decide on a transport mechanism for data (bit torrent, scp, or ?). We must take into consideration whether the data source matters in this choice (e.g. tape vs known good node), as well as how to identify that a data source is the correct one (e.g. via checksums and sequence numbers). | 12 |
| 1061 | DM-1998 | Architecture for failure detection and resolution | Come up with an architecture for detecting failure or non-nominal conditions (e.g. under replication). The core question to resolve is whether we go with a distributed approach, or with centralized control. | 16 |
| 1062 | DM-1999 | Research existing theory and prior art | Peruse the distributed systems literature for prior approaches to this problem, and examine existing system implementations for components/ideas we could reuse. | 10 |
| 1063 | DM-2000 | Document data distribution/replication plan | Produce an overview document that explains our definitions, architecture and strategies for dealing with data distribution and replication. | 10 |
| 1064 | DM-2001 | Define strategy for adding and removing data | Define a strategy (push/pull distributed/decentralized) for recognizing incoming data and cleaning up/removing stale/deleted data. Is data ingest just another form of failure recovery? | 10 |
| 1065 | DM-2002 | Define data distribution/replication testing strategy | Once we decide on a design for data distribution / replication, we should come up with a test plan. | 4 |
| 1066 | DM-2003 | Package Reorganization (Science Pipelines) | Breakdown: jbosch 50%, swinbank 50% | 38 |
| 1067 | DM-2005 | switch ndarray to external package | There is already an external ndarray project on GitHub (we've been using a fork of that). We should merge the forks and switch to using the external package. | 2 |
| 1068 | DM-2006 | merge "basics" packages | Create detailed RFC and implement merge of base, utils, and daf_base. See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning | 2 |
| 1069 | DM-2007 | separate pex_exceptions from base and rename | Remove dependency on base from pex_exceptions and rename to just "exceptions" (after RFC). See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning | 1 |
| 1070 | DM-2008 | Move Wcs from afw::image to afw::coord | Create RFC and implement move of Wcs from afw::image to afw::coord. See https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning | 1 |
| 1071 | DM-2010 | move Jarvis/shapelet code to legacy package | Create RFC and remove Jarvis/shapelet package from meas_algorithms, into new legacy sci package. See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning | 1 |
| 1072 | DM-2011 | move Psf, Kernel code to new afw::convolution subpackage | Create detailed RFC and implement move for these packages. See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning | 2 |
| 1073 | DM-2012 | split pipe_base into command-line and non-command-line components | Create detailed RFC and implement package split, to separate basic Tasks (to be used as e.g. subtasks) from CmdLineTask and ArgumentParser. See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning | 1 |
| 1074 | DM-2013 | Implement image stitching | This story involves implementing code that stitches images, simple case that does not involve tract boundaries. More advanced case in covered in separate ticket. We will need to determine WCS information for the target images. | 6 |
| 1075 | DM-2014 | Create interface and utility package for single-frame/forced processing | Create detailed RFC and implement move of interface and utility code from multiple existing packages to new package. See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning | 2 |
| 1076 | DM-2015 | Design and implement RESTful API for image stitching and rotation | NULL | 4 |
| 1077 | DM-2016 | Split PSF estimation and PSF model code into separate package | Create detailed RFC and implement move of concrete PSF estimation code and Psf subclasses from meas_algorithms to separate package. See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning | 1 |
| 1078 | DM-2017 | rename packages with minimal reorganization | meas_deblender, ip_isr, meas_astrom, meas_modelfit, and ip_diffim do not require major refactoring to fit into the new package reorganization, but they should be renamed (with sci_prefixes). See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning | 1 |
| 1079 | DM-2018 | Split measurement plugins into separate packages | Create detailed RFCs and implement splitting measurement plugins into separate package. May want one package for extremely basic plugins, always-used plugins (PixelFlags, TransformedCentroid, etc). See also https://confluence.lsstcorp.org/display/DM/Summer2015+Package+Reorganization+Planning | 2 |
| 1080 | DM-2019 | Split coaddition code and single-frame/forced command-line drivers | This should split all content in pipe_tasks into two packages (aside from what may have been removed in previous issues). | 2 |
| 1081 | DM-2020 | Research how to support L3 | Research implications of having to deal with updatable Level 3 data. | 12 |
| 1082 | DM-2021 | Architecture for supporting small non-partitioned tables | Some tables, like Exposure, provenance, will not be partitioned, and the current plan is to either replicate them on each node, or store on shared file system. Need to decide how it will be dealt with. | 6 |
| 1083 | DM-2022 | Research software deployment on the qserv cluster | Discuss and decide how to deploy and upgrade Qserv software | 10 |
| 1084 | DM-2023 | Investigate procedures for package reorganization | e.g. develop script to handle bulk namespace changes. | 4 |
| 1085 | DM-2024 | DRP S15 support for SQuaRE | Breakdown: jbosch 16%; lauren 16%; rhl 20%; pgee 16%; price 16%; swinbank 16% | 20 |
| 1086 | DM-2025 | FY15 Key Performance Metrics | Collect data, compile scripts, perform measurements as necessary to report figures in respect of the FY15 key performance metrics. Breakdown: lauren 50%; rhl 50% | 16 |
| 1087 | DM-2026 | Support Exposure Use Cases | Development in support of Exposure Use Cases | 19 |
| 1088 | DM-2027 | Implementing stitching multiple patches across tract boundaries in a coadd | NULL | 4 |
| 1089 | DM-2030 | refactor afw Swig to improve build times | I have an idea for how to improve Swig build times that we should get vetted (and possibly improved upon) by a true Swig expert (even if that costs a bit of $$). This issue includes vetting that idea (splitting up classes into multiple per-package module builds), getting it through the RFC process, and implementing it in afw. | 10 |
| 1090 | DM-2031 | Add image-query related KPIs to the plan | Existing plan in LDM-240 does not mention image related KPIs. Need to come up with a road map, and propose KPIs. This should be synchronized what realistically NCSA cluster can deliver in any given FY. | 5 |
| 1091 | DM-2034 | FY19 Setup Database for Deep Drilling | NULL | 53 |
| 1092 | DM-2035 | FY18 Setup Calibration Database | Need to think through issues related to supporting calibration. Schema, requirements that will require special optimizations. | 39 |
| 1093 | DM-2036 | S17 Build Prototype of AlertProd and L1 User Database | Build a working, non-optimized prototype of the [Alert Production and L1 User Database|http://ldm-135.readthedocs.org/en/master/#alert-production-and-up-to-date-catalog]. Deliverable: working, non-optimized prototype of AlertProd Database. | 79 |
| 1094 | DM-2037 | X16 Revisit Design of AlertProd & L1 Db | Revisit the [design of Alert Production Database and Level 1 User database|http://ldm-135.readthedocs.org/en/master/#alert-production-and-up-to-date-catalog], including schema, indexing, partitioning, synchronization, replicating, fail over. Verify that the latest requirements match the design. This epic will likely involve experimenting and light-weight standalone prototyping related to parts of the system that might be non-trivial to scale or to implement. Deliverable: a refreshed design document for Alert Production and L1 User Database. | 19 |
| 1095 | DM-2038 | FY17 Design Internal DRP Database | Internal DRP DB will be used to store * all bookkeeping (provenance, what run what did not, etc) * intermediate data products (might be larger than final data products), * a subset of data (what we need by DRP), eg foorprints of objects Internal DRP DB might need its own spatial engine. It is expected that SDQA will run on that database. Need to think through issues related to supporting internal Data Release Production. Schema, requirements that will require special optimizations. Need to define reliability requirements. In limited cases pipelines might want to use internal db (instead of files). Example: select all objects from a given region. Need to understand query load and complexity coming from DRP. | 79 |
| 1096 | DM-2039 | FY18 Revisit Design of L3 Support in Qserv | Need to think through issues related to supporting Level 3 databases | 100 |
| 1097 | DM-2040 | FY19 Design Next-to-database Data Analysis System | Need to design the system that will allow users run their own custom data analysis next to database. | 100 |
| 1098 | DM-2041 | FY19 Implement Next-to-database Data Analysis | Need to implement the system that will allow users run their own custom data analysis next to database. | 100 |
| 1099 | DM-2042 | W16 Improve Data Provenance Design | We have a detailed design of the Provenance, described at https://dev.lsstcorp.org/trac/wiki/db/Provenance. Work covered by this epic involves: 1. Revisiting the design and tweaking it as necessary. In particular: * Describing in more details interactions with key data producers (DRP, AlertProd, Calibration, L3 data brought in by users). * Estimating the size of provenance data * Considering querying the provenance data 2. Evaluating existing off-the-shelf provenance systems/tools. Deliverable: a document describing data provenance architecture / schema supported by a standalone proof-of-concept prototype. | 69 |
| 1100 | DM-2044 | Catch all epic for essential fixes in Science Pipelines DM-W15-5 | NULL | 10 |
| 1101 | DM-2045 | server side preparation for histogram plot (1) | Convert necessary code to make it possible for a JavaScript component to place a JSON request to the server and to parse the resulting RawDataSet. | 6 |
| 1102 | DM-2046 | Client side plot display for histogram | - Create a React JavaScript component, which takes the data and renders histogram. - Make it possible to call this component from GWT code, using experimental JsInterop technology in GWT 2.7 | 10 |
| 1103 | DM-2047 | SUI Investigate L3 data/tools requirements, evaluate potential tools | There are many overlap areas in L3 data analysis tools with the general science user tools. We want to identify those requirements and needs to help making SUI components adaptable for L3 data production and analysis. | 30 |
| 1104 | DM-2048 | Start requirements gathering for pipeline QA visualization needs | Gather use cases for pipeline QA visualization tools. We want to build the SUI components in such a way that they could be used to support QA needs. | 36 |
| 1105 | DM-2049 | SUI Build the visualization components that could be used independently | Currently we identified three basic components: Image visualizer, tabular data display, 2D XY Plot. All three could share the data model and provide inter activities between the components. | 80 |
| 1106 | DM-2050 | Integration and test monitoring architecture Part I | [retitled to better capture cycle scope] Develop and deploy a layer to capture the outputs, initially numeric, of integration testing afterburners such as sdss_demo, hsc_demo, and others developed this cycle. Also capture meta-information such as execution time and memory footprint. Propose log format to standardise production of such informations. Investigate notification system based on trending away from expected values. Investigate data provisioning of integration tests such as storage of test data in GithubLFS. [75% JMP 25% JH] | 100 |
| 1107 | DM-2051 | Firefly-based data display for SQuaSH - Part I | [Epic retitled to better reflect cycle scope] This epic covers work relating to working on the visualisation side of the Science QA Analysis Harness. It is a timeboxed effort to come up to speed with Firefly in particular, evaluate it against our needs, and provide any feedback to the Firefly team. Some prototyping of visualising integration dataset products will also be involved. [AF 100%] | 45 |
| 1108 | DM-2052 | Maintain list of OSes that pass build and integration testing | Provide an automatiically generated and updated pages showing operating systems that are successfully building and integrating the stack from source. [FE at 75%, JH at 75%] | 20 |
| 1109 | DM-2053 | Specify system for performing CI on Docker stack containers | Investigate how we can CI first-party Docker containers with runnable stacks [JH 100%] | 30 |
| 1110 | DM-2054 | Release engineering Part One | Bucket for public stack releases [FE at 75%, JH at 75%] | 40 |
| 1111 | DM-2055 | Miscellaneous service support improvements | JIRA, comm toos, etc for DM and non-DM teams (indicate) In order to avoid fractional story points, some 0 [FE at 75%, JH at 75%] | 16 |
| 1112 | DM-2057 | Attend Scale 13x conference | Attend database talks, in particular the MaxScale proxy talk (http://www.socallinuxexpo.org/scale/13x/presentations/advanced-query-routing-and-proxying-maxscale?utm_campaign=north-american-trade-shows&utm_source=hs_email&utm_medium=email&utm_content=16099082&_hsenc=p2ANqtz-_MFjfxvpCdmV_Ax2RKDdOGypHPQ85UL-UMuy0eRs_MrlJ2qJVp-MXx-g7_-dAQsq0trpA61hkZrzO-3gp6bKVkpK52fQ&_hsmi=16099082). If anyone has questions they would like me to ask, please post them here as well. I will post notes to this issue. | 2 |
| 1113 | DM-2058 | Data loader should always create overlap tables | We have discovered that some overlap tables that are supposed to exist were not actually created. It looks like partitioner is not creating overlap files when there is no overlap data and loader is not creating overlap table if there is no input file. Situation is actually symmetric, there could be non-empty overlap table but empty/missing chunk table. When we create one table we should always make another as well. | 2 |
| 1114 | DM-2059 | Clean up QuerySession-related code in czar | (created in response to DM-211) This ticket should address the following inelegancies in the qserv-czar. * QuerySession->QueryPipeline. The "session" abstraction has moved to a better place. The iterator portion should be shifted into its own separate class, though perhaps still associated with QueryPipeline. The iterator portion's new home should be amenable to eventually moving the actual query materialization to the worker, though we shouldn't introduce new abstractions until we are actually ready to move the substitution/materialization to the worker. * QueryContext needs to be split into incoming external QueryContext and a sort of QueryClipboard for passing information between analysis/manipulation plugins. Eventually, I imagine a chain/tree of them attached to the select statements themselves in order to represent subquery scope nesting (which is complicated to represent and to reason about--nesting and the resulting namespace resolution is tricky), but I don't think we should try doing the chaining in the first phase. For this ticket, create QueryClipboard to hold the portion for interchange between analysis plugins. Query analysis plugins would then pass this object (which points at an immutable? QueryContext) between themselves. QueryClipboard probably should live in qana, QueryContext in query (unless there is a good reason to move it). | 8 |
| 1115 | DM-2060 | Rename TaskMsgFactory2 | Rename TaskMsgFactory2 to TaskMsgFactory. Please see DM-211 for more information. | 1 |
| 1116 | DM-2061 | Port fault-recovery testing code to XrdSsi | Please see DM-211 for the origin of this ticket. BillC put in code to introduce random errors in query dispatch as part of working on code to recover from faults. In the port to the XrdSsi API, we did not port this code. This story is to introduce the ability (compile-time configurable, if not command-line or dynamically configurable) to simulate these sorts of faults to exercise the fault-recovery (confined to retrying on transient-ish failures) code. | 12 |
| 1117 | DM-2063 | Creates overlap tables even if empty while loading data | Query execution expects all chunk and overlap tables to exist, even if they are empty. In short term, that means loader should: * look at all chunks and add corresponding overlap chunks, * look at overlap chunks and add missing empty chunk table | 2 |
| 1118 | DM-2065 | FY17 Implement Data Verification Tool | Need a tool for verifying whether data is in consistent stage (e.g., right after loading, after some upgrades, in general at any given time). The list of things to check include: * empty chunk file, * xrootd exported DB, * data tables * overlap tables, * data_0123456789 tables * chunkId, subChunkId columns existence Some of the above can be automatically fixed on the spot when problem is discovered. | 90 |
| 1119 | DM-2066 | Add test case to catch missing empty chunks or overlaps | Discussed at db hangout 2015-02-18. We need a use case to test for missing empty overlap chunk tables and/or empty chunk tables. | 2 |
| 1120 | DM-2069 | FY17 Design L2 Catalog Swap/Release Automation | Need to think through the issues related to releasing L2 catalog / swapping a new one in place of an old one | 54 |
| 1121 | DM-2070 | FY17 Build AP-ready Data Provenance System | Improvements to the first version of the standalone prototype built through DM-2042. Discussions with the Application Team on capturing provenance and integrating DRP with the provenance system. Add scaffolding / unit tests that will simulate data producers, in particular DRP. Deliverable: DRP-ready System for capturing provenance. | 79 |
| 1122 | DM-2071 | FY18 Integrate AlertProd with Data Provenance | Integrate Alert Production with the Provenance system. | 56 |
| 1123 | DM-2072 | FY17 Implement Async Queries in Data Access Web Services | Work includes: * asynchronous requests, request status, retrieving results for dbserv and imgserv | 26 |
| 1124 | DM-2073 | FY19 Implement Partial Query Results | NULL | 100 |
| 1125 | DM-2075 | S17 Improve ImageServ | NULL | 9 |
| 1126 | DM-2077 | W16 Add Support for Multi-table Shared Scans | Implement multi-table shared scans. Ensure that shared-scans are not delaying interactive queries. The baseline architecture of the shares scans are described in [LDM-135 Shared Scanning|http://ldm-135.readthedocs.org/en/master/#shared-scanning]. | 100 |
| 1127 | DM-2078 | F16 Qserv KPMs | NULL | 24 |
| 1128 | DM-2079 | F17 Run Large Scale Qserv Tests | NULL | 26 |
| 1129 | DM-2080 | F18 Run Large Scale Qserv Tests | NULL | 26 |
| 1130 | DM-2081 | FY19 Implement Missing Features in Qserv for L3 | Need to think through issues related to supporting Level 3 databases | 100 |
| 1131 | DM-2082 | FY20 Improve Design of Next-to-database Data Analysis | Need to implement the system that will allow users run their own custom data analysis next to database. | 100 |
| 1132 | DM-2083 | FY18 Demonstrate Fault Tolerance | NULL | 100 |
| 1133 | DM-2084 | FY18 Implement Basic Resource Mgmt for DB | Includes things like query throttling per user for all databases (L1, L2, L3) | 54 |
| 1134 | DM-2085 | FY19 Optimize Resource Mgmt for DB | NULL | 79 |
| 1135 | DM-2086 | FY18 Implement Basic Resource Mgmt for Images | NULL | 54 |
| 1136 | DM-2087 | FY19 Optimize Resource Mgmt for Images | NULL | 79 |
| 1137 | DM-2088 | W16 Distributed Loader - Research | In production, we will need a distributed loader that will be capable of loading entire data set produced by DRP within 24-48 hours. This epic involves researching all the needs, requirements and constraints, and exploring what the best architecture for a distributed loader would be. Related doc: [LDM-135 §8.15.2|http://ldm-135.readthedocs.org/en/master/#data-loading] | 28 |
| 1138 | DM-2089 | W16 Distribution and Replica Mgmt Prototype v2 | This epic involves building a complete, working prototype of the Qserv data distribution and replica management. | 100 |
| 1139 | DM-2090 | FY18 Implement L2 Catalog Swap/Release Automation | Need to think through the issues related to releasing L2 catalog / swapping a new one in place of an old one | 100 |
| 1140 | DM-2091 | FY17 Add Support for Managing Per-user Access for DB | NULL | 53 |
| 1141 | DM-2092 | FY18 Add Support for Managing Per-user Access for Image and File Archive | NULL | 79 |
| 1142 | DM-2093 | FY18 Integrate Qserv with EFD | NULL | 79 |
| 1143 | DM-2094 | Port metaREST.py to db | metaREST_v0.py in metaserv is currently using MySQLdb instead of going through the db API, because we need to use parameter binding for security reasons. We should switch to using db, once the db interfaces will support it. | 1 |
| 1144 | DM-2095 | Port dbREST.py to db | dbREST_v0.py in dbserv is currently using MySQLdb instead of going through the db API, because we need to use parameter binding for security reasons. We should switch to using db, once the db interfaces will support it. | 1 |
| 1145 | DM-2096 | Long term database work planning | Long term planning (updating LDM-240). | 8 |
| 1146 | DM-2097 | Package andyH xssi fixed version (>2MB answer pb) in eups | See DM-1847 - Andy made a patch, it'd be good to the xrootd we use for our stack. | 1 |
| 1147 | DM-2101 | FY18 Revisit L2 Catalog Schema | Revisit the baseline schema | 53 |
| 1148 | DM-2103 | FY18 Implement Internal DRP Database | Implement Internal DRP Database as designed in DM-2038 | 60 |
| 1149 | DM-2107 | FY17 Improve ImageServ | NULL | 80 |
| 1150 | DM-2108 | FY19 Demonstrate Qserv Fault Tolerance | Including multi-master failover | 100 |
| 1151 | DM-2110 | FY19 Optimize Partitioning Granularity | We have been always talking about having ~20K chunks per table, and it was driven primarily by spreadsheet-based analysis. We need to look in more details into that, and perhaps even change the model if needed, e.g., introduce different partitioning for larger tables, like ForcedSource. | 53 |
| 1152 | DM-2111 | FY17 Improve Query Coverage in Qserv | Currently Qserv supports only a limited subset of queries. We need to make sure it supports all queries that users need to run. | 53 |
| 1153 | DM-2112 | FY18 Improve Query Coverage in Qserv | Currently Qserv supports only a limited subset of queries. We need to make sure it supports all queries that users need to run. | 79 |
| 1154 | DM-2113 | FY17 Support Explain, Show, List Commands | Implement [explain|http://dev.mysql.com/doc/refman/5.0/en/explain.html] and [show|http://dev.mysql.com/doc/refman/5.0/en/show.html] commands for Qserv. Also, commands such as "list tables" will need to be intercepted and overloaded. | 53 |
| 1155 | DM-2115 | FY19 Implement Multi-master for Qserv | NULL | 100 |
| 1156 | DM-2116 | FY19 Make Database Secure | Revisit security issues, such as sql injections, detecting and shielding from DoS attacks, etc. | 79 |
| 1157 | DM-2117 | FY19 Build/Setup Basic Qserv Monitoring | frontend/worker health monitoring (and management?) | 79 |
| 1158 | DM-2118 | FY18 Implement Failover for L1 Database | Need to implement and test failover - a failure of the master copy of L1 database, and automatic fail over to a replica. The design of the Alert Production L1 database is covered [here|http://ldm-135.readthedocs.org/en/master/#alert-production-and-up-to-date-catalog]. | 80 |
| 1159 | DM-2119 | W16 Optimize Secondary Index - Research | Work on the secondary index (objectId --> chunkId / subChunkId mapping). This needs to be scalable to 40B entries. Since we are planning to ingest all data from DRP in <2 days, building should take <2 days. This epic involves researching applicable technologies (including experimenting with most promising ones). Deliverable: proposed technology / architecture along with measures performance at production scale (40 B entries). | 48 |
| 1160 | DM-2123 | FY18 Add Support for Non-partitioned Tables | Non partitioned tables will need special attention. Options include: a. replicating them on each worker node b. keeping them on a shared file system c. federating Need to thing through these issues, pick the best architecture and implement it. | 79 |
| 1161 | DM-2124 | FY17 Revisit Qserv Deployment on Cluster | NULL | 60 |
| 1162 | DM-2125 | FY18 Design Qserv Software Upgrading | Need to understand how to do software update for Qserv | 26 |
| 1163 | DM-2129 | FY19 Improve Query Coverage in Qserv | NULL | 90 |
| 1164 | DM-2130 | FY20 Improve Qserv Monitoring | frontend/worker health monitoring (and management?) | 79 |
| 1165 | DM-2131 | Resolve compiler warnings in new measurement framework | When building {{meas_base}}, or any other measurement plugins which follow the same interface, with clang, I see a bunch of warnings along the lines of: {code} In file included from src/ApertureFlux.cc:34: include/lsst/meas/base/ApertureFlux.h:197:18: warning: 'lsst::meas::base::ApertureFluxAlgorithm::measure' hides overloaded virtual function [-Woverloaded-virtual] virtual void measure( ^ include/lsst/meas/base/Algorithm.h:183:18: note: hidden overloaded virtual function 'lsst::meas::base::SimpleAlgorithm::measure' declared here: different number of parameters (4 vs 2) virtual void measure( {code} This is an artefact of a [workaround for SWIG issues|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=20284390]; the warnings aren't indicative of a fundamental problem, but if we can avoid them we should. While we're at it, we should also fix: {code} include/lsst/meas/base/ApertureFlux.h:233:1: warning: 'ApertureFluxResult' defined as a struct here but previously declared as a class [-Wmismatched-tags] struct ApertureFluxResult : public FluxResult { ^ include/lsst/meas/base/ApertureFlux.h:65:1: note: did you mean struct here? class ApertureFluxResult; ^~~~~ struct {code} | 1 |
| 1166 | DM-2136 | W16 Understand Async Queries in Qserv | Understand how disruptive the changes related to implementing asynchronous queries will be for Qserv. Delivarable: brief description outlining changes needed, with story point estimate. | 10 |
| 1167 | DM-2137 | Add parameter binding to db interface | NULL | 1 |
| 1168 | DM-2138 | Validate user query in dbREST | Need to validate query (from security standpoint that user enters through rest api. | 1 |
| 1169 | DM-2139 | Support DDL in MetaServ - implementation | DDL information is embedded as comments in the master version of the schema (in "cat" repo). Currently we are only using it for schema browser. This story involves building tools that will load the DDL schema into MetaServ. Design aspects are covered in DM-1770. | 8 |
| 1170 | DM-2141 | Add meas_extensions_shapeHSM to lsstsw, lsst_distrib | meas_extensions_shapeHSM has just been resurrected from bitrot, and should be included in our distribution. Contrary to DM-2140, it should probably not be included in lsst_apps, as it's not clear we want to add a dependency on tmv and GalSim there. | 1 |
| 1171 | DM-2148 | General OpenStack Learning | NULL | 6 |
| 1172 | DM-2149 | Setup spare test hardware for OpenStack testing | NULL | 20 |
| 1173 | DM-2150 | Test Ubuntu OpenStack | NULL | 6 |
| 1174 | DM-2151 | Test Mirantis OpenStack & Fuel | NULL | 6 |
| 1175 | DM-2152 | Figure out OpenStack networking (vLAN, routing, etc) | NULL | 6 |
| 1176 | DM-2153 | Figure out OpenStack integration with LDAP | NULL | 17 |
| 1177 | DM-2155 | Log fails on uniccode string | Log is currently failing if we pass unicode string, it is easy to reproduce by doing: log.info(u"hello"). It fails with: {code} File "/home/becla/dataArchDev/Linux64/log/master-gfab0203bd3+3/python/lsst/log/log.py", line 103, in info log("", INFO, fmt, *args, depth=2) File "/home/becla/dataArchDev/Linux64/log/master-gfab0203bd3+3/python/lsst/log/log.py", line 94, in log _getFuncName(depth), frame.f_lineno, fmt % args) File "/home/becla/dataArchDev/Linux64/log/master-gfab0203bd3+3/python/lsst/log/logLib.py", line 648, in forcedLog_iface return _logLib.forcedLog_iface(*args) TypeError: in method 'forcedLog_iface', argument 6 of type 'std::string const &' {code} | 1 |
| 1178 | DM-2157 | Data loader crashes on uncompressed data. | Vaikunth just mentioned to me that the is a crash in data loader when it tries to load uncompressed data: {noformat} root - CRITICAL - Exception occured: local variable 'outfile' referenced before assignment Traceback (most recent call last): File "/home/vaikunth/src/qserv/bin/qserv-data-loader.py", line 312, in <module> sys.exit(loader.run()) File "/home/vaikunth/src/qserv/bin/qserv-data-loader.py", line 248, in run self.loader.load(self.args.database, self.args.table, self.args.schema, self.args.data) File "/home/vaikunth/src/qserv/lib/python/lsst/qserv/admin/dataLoader.py", line 168, in load return self._run(d atabase, table, schema, data) File "/home/vaikunth/src/qserv/lib/python/lsst/qserv/admin/dataLoader.py", line 192, in _run files = self._gunzip(data) File "/home/vaikunth/src/qserv/lib/python/lsst/qserv/admin/dataLoader.py", line 388, in _gunzip result.append(outfile) UnboundLocalError: local variable 'outfile' referenced before assignment {noformat} It looks like we never tested loader on uncompressed data and there is a bug in handling uncompressed data. | 1 |
| 1179 | DM-2158 | Add support for JSON - define structure | As discussed at [Data Access Hangout 2015-02-23|https://confluence.lsstcorp.org/display/DM/Data+Access+Hangout+2015-02-23], we should support json format. This includes defining the exact format, and implementing it. This story covers defining the format. | 2 |
| 1180 | DM-2159 | Implement Image Response for ImgServ | This story covers implementing proper response, and the header metadata for the fits image response. | 3 |
| 1181 | DM-2161 | Setup webserv for SUI tests | We need to setup a service (eg on lsst-dev) that can be used by the IPAC team to play with our webserv/metaserv/dbserv/imgserv. The server runs on lsst-dev machine, port 5000. To ssh-tunnel, try: {code} ssh -L 5000:localhost:5000 lsst-dev.ncsa.illinois.edu {code} An example usage: {code} curl 'http://localhost:5000/db/v0/query?sql=SHOW+DATABASES+LIKE+"%Stripe%"' curl 'http://localhost:5000/db/v0/query?sql=SHOW+TABLES+IN+DC_W13_Stripe82' curl 'http://localhost:5000/db/v0/query?sql=DESCRIBE+DC_W13_Stripe82.DeepForcedSource' curl 'http://localhost:5000/db/v0/query?sql=DESCRIBE+DC_W13_Stripe82.Science_Ccd_Exposure' curl 'http://localhost:5000/db/v0/query?sql=SELECT+deepForcedSourceId,scienceCcdExposureId+FROM+DC_W13_Stripe82.DeepForcedSource+LIMIT+10' curl 'http://localhost:5000/db/v0/query?sql=SELECT+ra,decl,filterName+FROM+DC_W13_Stripe82.Science_Ccd_Exposure+WHERE+scienceCcdExposureId=125230127' curl 'http://localhost:5000/image/v0/raw/cutout?ra=7.90481567257&dec=-0.299951669961&filter=r&width=30.0&height=45.0' {code} | 2 |
| 1182 | DM-2163 | Refactor Geom class in Firefly | The Geom class was ported from C code 20 years ago. It needs to refactor to comply with Java OO design. | 8 |
| 1183 | DM-2164 | Review at DM leadership team meeting | review document with Kantor, KT, Hobblit, and Lambert, including prep time | 3 |
| 1184 | DM-2165 | Refactor document for that specifications are clearer | 1) Have one basic definition of racks and other components in the specifications. 2) Fully write up first full draft specification for the supporting material handing area. | 3 |
| 1185 | DM-2166 | receive / process comments from Jeff Barr a | receive edits from Jeff Barr, accept the formatting and mechanical l edits. Compose separate email list issues related to non LSST tenants in the room. | 1 |
| 1186 | DM-2167 | Investigate Commerical vendor to deal with comments on requirements. | process email discussion about the need to liaison with the putative Chilean design contractor. Kantor suggests a contractor to support requirements may be apropos. | 1 |
| 1187 | DM-2168 | Work inside NCSA to connect procurement contract Modification to OSPRA contract officet | work Jeff's proposal until it reached university contract officer. -- January meeting -- clarify purchasing rules - Internal discussion of property management, - General work within contract modification process. | 6 |
| 1188 | DM-2171 | Implement JSON Results for MetaServ and DbServ | Implement JSON results for Metadata Service (see all M* in https://confluence.lsstcorp.org/display/DM/API), and Database Service (see all D*) as defined in DM-1868 | 3 |
| 1189 | DM-2173 | Disable testDbLocal.py in db if auth file not found | tests/testDbLocal.py can easily fail if required mysql authorization file is not found in user home dir. Skip the test instead of failing in such case. | 1 |
| 1190 | DM-2175 | Adapt integration test to multi-node setup v2 | Following DM-595 we can start qserv in multi-node configuration. Next step is to be able to run integration tests in that setup. This needs a bit of understanding how to distribute chunks between all workers in a cluster and how to load data in remote mysql server. | 10 |
| 1191 | DM-2176 | Worker management service - impl | We need to replace direct worker-mysql communication and other administrative channels with a special service which will control all worker communication. Some light-weight service running alongside other worker servers, probably HTTP-based. Data loading, start/stop should be handled by this service. | 10 |
| 1192 | DM-2177 | Implement worker-side squashing | In the port to the new Xrootd Ssi API, worker-side squashing was lost in the shuffle. The plumbing is different, and re-implementing squash functionality is not entirely straightforward, especially because the new API is still missing documentation and examples for implementing cancellation. The consequences of not implementing this are minor--some extra work may be done by the worker, but not a whole lot, because user-level cancellation has not been implemented. | 12 |
| 1193 | DM-2178 | Migrate Qserv to external sphgeom | Migrating qserv to the new c++ geometry API required porting a fair amount of code from the python layer and updating the plumbing in the czar. During implementation, the sphgeom was in the process of finding a home, so the sg code was temporarily placed under core/modules. This ticket covers: * removing core/modules/sg * updating code to point at the external sphgeom * updating build-logic to properly depend on and link with external sphgeom. | 4 |
| 1194 | DM-2181 | S17 Design Prototype EFD Schema for DRP | The epic involves understanding the structure of the EFD database produced by the Engineering and Facility team, and designing schema that will be best suited for Data Release Production. Note that the original EFD database may not even be in MySQL, there were discussions to store it in Postgresql. Deliverable: Alpha version of the EFD database schema for DRP with "real" data loaded (if available). | 52 |
| 1195 | DM-2182 | FY18 Design DRP-ready EFD Schema | NULL | 79 |
| 1196 | DM-2186 | Move astrometry_net wrapper code from meas_astrom to meas_astrometry_net | We would like to remove all astrometry.net wrapper code from meas_astrom and put it in a new package with a name such as meas_astrometry_net. This will also require moving any abstract task base classes into a lower-level package such as meas_astrom. | 6 |
| 1197 | DM-2187 | FY17 Data Loader for Large Tables with No Position Information | We need to load some tables (e.g., ForcedSource) that lack director positioning, we will only have the director's primary key. The general case is very expensive (lookup position and chunk for each position), however the fact such tables will be spatially-ordered when loading helps. | 60 |
| 1198 | DM-2188 | Update the astrometry.net astrometry solver to use the new standard schema | DM-1576 provides a new astrometry solver and a new schema for reference objects. However, the old astrometry.net astrometry solver still uses the old schema. It would be wise to convert the old solver to the new schema so that the match list returned by it is in standard format. | 4 |
| 1199 | DM-2189 | Large scale test planning | Need to come up with a plan which data set we will use for large scale tests, and how we will produce it. | 10 |
| 1200 | DM-2190 | Documentation for data loader | Vaikunth had some "expected" troubles playing with data loader options for his DM-1570 ticket. Main issue I believe is the absence of the documented use cases and their corresponding data loader options. I'll try to add a bunch of common use cases to RST documentation and also verify that all options behave as expected. | 2 |
| 1201 | DM-2191 | Define command line tasks for pre-ingest transformation | DM-1903 provided a command line task which would transform a {{src}} catalogue into calibrated form. Here, we build on that to provide command line tasks for all source catalogues which will need to be ingested; will include at least {{deepCoadd_src}}, {{goodSeeingCoadd_src}}, {{chiSquaredCoadd_src}}. | 6 |
| 1202 | DM-2192 | Provide transformations for "big three" measurements | Provide standard calibration transformations for each of shape, flux and centroid and make sure they are returned as the default transformation for all algorithms measuring those quantities. | 10 |
| 1203 | DM-2193 | Add assertXNearlyEqual to afw | We often want to compare two WCS for approximate equality. afw/image/testUtils has similar functions to compare images and masks and I would like to add one for WCS This ended up being expanded to adding functions for many afw classes (not yet including image-like classes, though existing functions in image/testUtils for that purpose should probably be wrapped or rewritten on a different ticket) | 5 |
| 1204 | DM-2194 | Ensure proper functioning of HSC distortion correction within obs_subaru | There may be some discrepancy between the pixel units being passed to distest.cc compared to what it is expecting (units of pixels). This needs to be investigated further and remedied in such a way that all other representations (e.g. in camera.py) are consistent with the other obs_XXX representations. | 6 |
| 1205 | DM-2195 | Create form framework in React | We want to create a new frame work for entering data for forms and dialogs. This is in javascript based on React.js. This is the first step in our javascript conversion. | 10 |
| 1206 | DM-2197 | Prototype HTM-based spatial binning to visualize large number of catalog sources | See story DM-1551. | 8 |
| 1207 | DM-2199 | Build 2015_03 Qserv release | See https://confluence.lsstcorp.org/display/DM/Qserv+Release+Procedure for recipe. | 1 |
| 1208 | DM-2201 | Add typemaps for numpy scalars | Add typemaps so that we can use numpy scalars to call C++ functions that take plain old scalar types (e.g. float, double or int). At present attempting to pass numpy scalars will fail unless the type is one of a restricted subset, e.g. float and numpy.float64 succeed but numpy.float32 is rejected as being an incompatible type, and similarly for integer types. | 4 |
| 1209 | DM-2202 | Acquire development data | We'll need a reference set of data to work against. This could be SDSS, CFHT, or simulated. Should be 10? epochs with realistic atmospheric conditions taken at similar airmass and hour angle. Single band is fine for now. | 4 |
| 1210 | DM-2203 | Produce task API | This will require a new task, so will require a new interface and associated RFCs. The interface should take an arbitrarily large stack of catalogs with or without a reference catalog. It should return a stack ow WCSs that map from the individual coordinate systems to the reference. | 6 |
| 1211 | DM-2205 | Break down monster DM-1108 stories | [~pgee] -- After finishing the measurement work, your next priority is to get started on DM-1108. However, the stories you have been assigned there are currently too big for useful scheduling (20-30 SPs is a mini-epic; we're looking for less than 10 SPs per story). The first task therefore is to work with [~jbosch], and others if required, to break them down and come up with a set of stories which usefully reflect the work which needs to be done. | 4 |
| 1212 | DM-2206 | Deploy and test network emulation for nightly processing testbed | Deploy and test network emulation for nightly processing testbed. Assignees: Paul Wefel, Steve Pietrowicz, James Parsons Duration: January - February 2016 | 19 |
| 1213 | DM-2207 | Alert Production Simulator | Start March 2015, finish July 2015 Pietrowicz S - 100% | 96 |
| 1214 | DM-2208 | Complex Event Processing | Start May 2015, finish June 2015 | 31 |
| 1215 | DM-2209 | OCS Software | Start July 2015, finish August 2015 Pietrowicz S - 100% | 10 |
| 1216 | DM-2210 | Configuration Management (Puppet) | Start March 2015, finish May 2015 Mather B - 40% | 9 |
| 1217 | DM-2211 | Setup qserv prototype for qserv & SUI teams | Start July 2015, finish August 2015 Glick B - 25% Qserv requirements: - SUI will be testing against lsst10 (or IPAC qserv) for now ? SUI requirements: Xiuqin's 'short term' version: - 1 VM - SUI build server - 4GB memory and 200GB hard disk should be good enough. - 1 VM - Apache server as a proxy and web front end - 4GB memory and 100GB hard disk should be enough - port 80 accessible from outside - 2 VMs - Tomcat servers - each has 16GB memory, access to 1TB of shared hard disk - Port 8080 should be open for Apache server to access - Port 8009 should be open to each other so they can replicate cache. (First 2 VMs are not absolutely needed. We can always use one of the hosts in number 3 to do build and host Apache server.) Trey Roby's 'long term' (in 2+ years) SUI requirements: - 2 vm/machines for Tomcat servers, they are fairly large - each 100 GB mem - each 16 processors - 1 TB disk space shared and accessible between. - 1 vm/machine for Web Server, can be small - 16 GB mem - 30 GB disk - 4 processors - 2 small vm/machines for playing around with workspaces/L3 concepts - each has 8 GB mem - each 10 GB disk - 4 processors - Can share the Tomcat servers disk space | 10 |
| 1218 | DM-2213 | Storage Policies and Alignment | Start March 2015, finish August 2015 Freemon M - 100% | 50 |
| 1219 | DM-2214 | File System Research and Prototyping | Start March 2015, finish September 2015 Freemon M - 100%, Glick B - 25%, Daues G - 40%, Elliot M - 25% | 100 |
| 1220 | DM-2215 | File Management Technology | Start March 2015, finish September 2015 Daues G - 40%, Freemon M - 100% | 50 |
| 1221 | DM-2216 | Understand GPFS and commercial filesystems between data centers | Start May 2015, finish June 2015 Petravick D - 50%, TBD from SET group | 6 |
| 1222 | DM-2217 | Update Sizing Model | Start March 2015, finish September 2015 Alt J - 50%, Petravick D - 10% | 22 |
| 1223 | DM-2218 | Base Data Center Requirements | Start March 2015, finish May 2015 Petravick D - 50% | 18 |
| 1224 | DM-2221 | Start understanding inheritability and reusability of dataset types | In order to allow for on-the-fly Task creation of dataset types, the essentials of each type need to be encapsulated in code. That code should be reused across all similar dataset types, and there are opportunities for inheritance and specialization, particularly in cases like simple file-oriented mappers. Investigate this by prototyping a number of possibilities. | 4 |
| 1225 | DM-2224 | Wide-Area Network Work | Start March 2015, finish September 2015 Wefel P - 25% | 20 |
| 1226 | DM-2225 | LOE - S15 (sys admin) | Glick B - 25%, Mather B - 40%, Elliot M - 25%, Freemon M - 100% Wefel P - 25% | 80 |
| 1227 | DM-2226 | LOE - S15 (management) | Petravick D - 50%, Gelman M - 50% | 20 |
| 1228 | DM-2227 | LOE - S15 (misc) | All NCSA team | 10 |
| 1229 | DM-2228 | LOE - Week ending 3/6/15 | - backup issues with lsst-stor141 (https://jira.ncsa.illinois.edu/browse/LSST-632) - setup jumbo frames on lsst-xfer (https://jira.ncsa.illinois.edu/browse/LSST-628) - crashplan reconfig on lsst-xfer (https://jira.ncsa.illinois.edu/browse/LSST-629) | 8 |
| 1230 | DM-2229 | LOE - Week ending 3/13/15 | - crashplan issue with lsst-netem (https://jira.ncsa.illinois.edu/browse/LSST-633) - yum/glibc issue with lsst-dbdev1 (https://jira.ncsa.illinois.edu/browse/LSST-631) - account for Jacques Sebag (https://jira.ncsa.illinois.edu/browse/LSST-624) | 8 |
| 1231 | DM-2230 | LOE - Week ending 3/20/15 | - lsst-dbdev2 drive failure (https://jira.ncsa.illinois.edu/browse/LSST-636) - account for Colin Slater (https://jira.ncsa.illinois.edu/browse/LSST-634) - disable Robyn Allsman's accounts (https://jira.ncsa.illinois.edu/browse/LSST-623) | 26 |
| 1232 | DM-2231 | LOE - Week ending 3/27/15 | NULL | 21 |
| 1233 | DM-2232 | LOE - Week ending 4/3/15 | - researched buildbot slowness on lsst-dev <https://jira.lsstcorp.org/browse/DM-2388> - researched buildbot slowness on lsst-dev <https://jira.ncsa.illinois.edu/browse/LSST-638> | 16 |
| 1234 | DM-2233 | LOE - Week ending 4/10/15 | NULL | 16 |
| 1235 | DM-2234 | LOE - Week ending 4/17/15 | NULL | 17 |
| 1236 | DM-2235 | LOE - Week ending 4/24/15 | - Researched how to monitor network drops, errors, etc <https://jira.ncsa.illinois.edu/browse/LSST-641> - Opened up SUI ports on lsst-dev <https://jira.ncsa.illinois.edu/browse/LSST-651> - Moved /nfs/admin/ to /condo/admin/ - Review of EL 6.x kernel security patch - Fixed jumbo frame issues, primarily with old VMs that needed new version of NIC <https://jira.ncsa.illinois.edu/browse/LSST-652> | 15 |
| 1237 | DM-2236 | Prototyping with Puppet | NULL | 5 |
| 1238 | DM-2237 | Test Puppet with base configuration manifests | NULL | 38 |
| 1239 | DM-2239 | Develop use cases for TOWG | Start March 2015, finish May 2015 Petravick D - 50%, Glick B - 25%, Gruendl R - 5% | 20 |
| 1240 | DM-2240 | ISO Work | Start March 2015, finish September 2015 Withers A - 25% | 40 |
| 1241 | DM-2243 | Extend API: expose cursor | Extend API to expose cursor. This was brought up by Andy in DM-2137. | 1 |
| 1242 | DM-2245 | Define ntermediate plan for MacOSX builds | We have 1. Obtain a dedicated colo OSX server 2. Have done some testing using the SQuaRE vagrant-sandbox harness It is therefore a plausible avenue forward to do at least a nightly build/deploy/intgeration-test on OSX pending more extensive arrangements requiring purchase of hardware. | 4 |
| 1243 | DM-2246 | Github transition for DM | DM's transition for code repositories to Github is complete. Outstanding are data repositories; a cleanup of contrib/externals; and supporting the Stash move. | 10 |
| 1244 | DM-2247 | Workflow improvements for Sims / PST projects | New wokflow for Sims Merge of Opsim and CATsim New workflow for PST | 5 |
| 1245 | DM-2248 | Prototype automated system for release preparation builds | Prototype an environment that allows automatic - Provisioning of a VM for a certain OS - Install the Stack prerequisites for that OS - Build the stack via newinstall.sh from the production server - Run integration tests (in the curent case the sdss test https://github.com/lsst-sqre/sandbox-stackbuild | 22 |
| 1246 | DM-2250 | Galaxy Fitting via "ngmix" | Provide wrappers that let us run Erin Sheldon's [ngmix|https://github.com/esheldon/ngmix] as part of the DM pipeline. Issues so far only cover getting the a single-frame (visit or coadd) version of the code running. ngmix can also to simultaneous fitting to multiple exposures, but it's not yet clear how we'll want to handle the I/O and that interacts with a future multifit plugin framework. | 45 |
| 1247 | DM-2251 | Implement SExtractor's SPREAD_MODEL | The new SExtractor star/galaxy classifier, SPREAD_MODEL, is popular with everyone who has tried it, and should be simple to implement by building on code in meas_modelfit. See definition and discussion here: http://arxiv.org/abs/1306.4446 | 2 |
| 1248 | DM-2252 | Define common interface for star/galaxy classifiers | We need some common fields for star/galaxy classifiers so they can participate in a slots-like mechanism once we have several of them. Most of these can produce a floating point number between 0 and 1 (but sometimes it's not limited to that range), and it's rarely a true probability. We may want to make a boolean that results from a threshold on these be the common interface, but we don't necessarily want to hard-code such a threshold into the processing either - especially when we could also use a FunctorKey to get a boolean from the floating point value. | 2 |
| 1249 | DM-2253 | add third-party package builds for ngmix dependencies | In addition to numpy and scipy, ngmix depends on the emcee and statsmodel packages. While it can build without them, we probably want the full functionality. I also see some undeclared dependencies on the "esutil" and "fitsio" packages (all from esheldon's GitHub), and there may be a few more dependencies on some of his own packages. This issue includes creating a third-party build for ngmix itself. | 6 |
| 1250 | DM-2254 | Add SFM plugin for ngmix MCMC sampling | Add an SFM plugin for ngmix MCMC fitting, as in the example in the ngmix README. This should depend on DM-5429 (or a suitably configured modelft_ShapeletPsfApprox) for PSF approximation. For now, we should just take the mean of all parameters in the MCMC samples and write those to the record, as we currently don't have any way to save all of the samples. Testing and tuning this algorithm to get it working well should be deferred to another issue. The only requirement here is that it be able to run without crashing (even if that means setting the number of samples small). | 10 |
| 1251 | DM-2256 | make a simple build for Firefly package | We want to have a out of box build for users of Firefly package. It will include a simple Firefly viewer. | 6 |
| 1252 | DM-2257 | Allow eups xrootd install script to be relocatable | xrootd lib/ directory should be s relative symlink to lib64, no a full path link. | 1 |
| 1253 | DM-2258 | Setup in2p3 cluster for Qserv team | - create accounts - update umask on stack to each account - provide easy ssh config if possible - setup up build procedure (each developer can build Qserv using tag git and 'git' version is set up by default on all the Qserv if ti exists) | 2 |
| 1254 | DM-2259 | remove PSFAttributes | PSFAttributes has long been deprecated, and we just need a little more work to remove it: - Add an effective area accessor to the Psf interface, and implement it in ImagePsf. - Replace usage of PSFAttributes with usage of Psf accessors. This may require a little work if code depends on the details of how the shape was calculated, as PSFAttributes provided support for more algorithms than we will going forward. | 2 |
| 1255 | DM-2262 | Improve build system for sphgeom | NULL | 2 |
| 1256 | DM-2263 | Create pilot condor jobs | Create long-running jobs to reduce the startup time for new HTCondor jobs. This can be implemented as a parent/child, or as a on_exit_remove=false directive in HTCondor. I suspect it will be a combination of the two. | 21 |
| 1257 | DM-2264 | Implement task switching between work job machines | AP requires that jobs are handed off to different worker job clusters as the previous set of images is being worked on. | 4 |
| 1258 | DM-2265 | Refactoring | The initial prototype of the AP simulator needs to be refactored to improve how tasks are handled by the components for further development. | 25 |
| 1259 | DM-2268 | Implement API for reading simulated camera data | Currently this is generated by the replicator and sent to the distributor. The idea where is to put the API in place so that the data will be transferred from outside of the replicator to it, and then passed on. | 10 |
| 1260 | DM-2269 | Implement file transfer API | Create file transfer API so we can easily test different types of file transfer mechanisms to/from the AP. | 4 |
| 1261 | DM-2270 | Move VMs to Docker containers | We anticipate being able to move from the VMs that we currently use to using docker. This will require some coordination with Greg Daues to see how HTCondor is configured. | 2 |
| 1262 | DM-2272 | Unify logging strategy for python scripts | - add -vvv option - remove default value for configuration file in logger, provide it at each script level (i.e. integration test, data loader). - if it exists, provide configuration file option explicitly to all called submodules which uses it. See admin/python/lsst/qserv/admin/logger.py {code:python} 14 def get_default_log_conf(): 15 default_log_conf = "{0}/.lsst/logging.ini".format(os.path.expanduser('~')) 16 return default_log_conf 17 18 def add_logfile_opt(parser): 19 """ 20 Add option to command line interface in order to set path to standar 21 configuration file for python logger 22 """ 23 24 parser.add_argument("-V", "--log-cfg", dest="log_conf", 25 default=get_default_log_conf(), 26 help="Absolute path to yaml file containing python" + 27 "logger standard configuration file") 28 return parser 29 30 31 def setup_logging(path='logging.ini', 32 default_level=logging.INFO): 33 """ 34 Setup logging configuration from yaml file 35 if the yaml file doesn't exists: 36 - return false 37 - configure logging to default_level 38 """ 39 if os.path.exists(path): 40 with open(path, 'r') as f: 41 logging.config.fileConfig(f) 42 return True 43 else: 44 logging.basicConfig(level=default_level) 45 return False {code} | 6 |
| 1263 | DM-2277 | Document HOW-TO setup-up krb5 for easy cluster access | {code:bash} su aptitude install krb5-user # edit /etc/krb5.conf w.r.t ccage one # then as desktop user kinit ssh ccqservxxx {code} /etc/krb5.conf {code:bash} [libdefaults] default_realm = IN2P3.FR ... allow_weak_crypto = true ... [realms] IN2P3.FR = { kdc = kerberos-1.in2p3.fr:88 kdc = kerberos-2.in2p3.fr:88 kdc = kerberos-3.in2p3.fr:88 master_kdc = kerberos-admin.in2p3.fr:88 admin_server = kerberos-admin.in2p3.fr kpasswd_server = kerberos-admin.in2p3.fr default_domain = in2p3.fr {code} sshconfig: {code:bash} Host ccqservbuild GSSAPIAuthentication yes GSSAPIDelegateCredentials yes ForwardX11 yes HostName ccqservbuild.in2p3.fr #ProxyCommand ssh -W %h:%p cc Host ccqserv1* GSSAPIAuthentication yes GSSAPIDelegateCredentials yes ForwardX11 yes HostName %h.in2p3.fr ProxyCommand ssh -W %h:%p ccqservbuild {code} | 2 |
| 1264 | DM-2279 | Fix problems with mysql timeout | We added some code for supporting reconnecting (see https://dev.lsstcorp.org/trac/ticket/3042) but clearly not enough to recover from connection timeouts. This needs to be addressed. | 1 |
| 1265 | DM-2280 | The TAN_PIXELS cameraGeom coordinate system should be with respect to the center of the focal plane | The TAN_PIXELS cameraGeom coordinate system (the position on a detector if there is no optical distortion) is presently defined with respect to the center of the detector -- i.e. a star at the center of the detector will have the same position in PIXELS and TAN_PIXELS coordinates. That is a mistake. TAN_PIXELS should be defined with respect to the center of the focal plane, since it then reflects the effects of having optical distortion or not. Fixing this will help meas_astrom match stars. The effects of not fixing it are making the matcher search farther for a fit. As long as we allow sufficient offset in the matcher config the current system will work, but it is not ideal. | 2 |
| 1266 | DM-2281 | Implement connection pool | Implement a class that manages a connection pool, and optionally, if configured, restarts connection as needed in case of timeout. | 1 |
| 1267 | DM-2282 | Switch to using db connection pool | Switch to using the db connection pool. Note, in addition to getting auto-reconnect, in metaserv that would handy if we need to talk to multiple database servers simultaneously. | 1 |
| 1268 | DM-2286 | Participate in design process | Participate and guide the SUI design process, generate charts and documents as appropriate | 9 |
| 1269 | DM-2287 | Move javascript code into firefly repo and begin creating a real input form | NULL | 10 |
| 1270 | DM-2288 | Work with Camera & Pipeline team to spec out proof of concept tools | NULL | 14 |
| 1271 | DM-2289 | Personnell requisitions | Work though recruiting for software effort. Investigated and filled the "kenton" recruiting pattern at NCSA -- (few explicit requirements, many desirable) Began discussion to break down hires for "2nd" floor work -- to be in the LSST group v.s support groups -- ADS and Doug's group | 3 |
| 1272 | DM-2290 | Arrange for commercial object store presentation | arrage for presentations next week w.r.t commercial object store. The vendor in question is know to NCSA and has claims to have produced a commercial object store having both NFS, GPFS and swift interfaces. | 1 |
| 1273 | DM-2291 | Begin WBS review | Begin comprehensive review of the WBS. Forced on overall framework and begin workflow systems | 1 |
| 1274 | DM-2292 | Security officer orientation | begin orientation of LSST ISO ALEX Withers. | 1 |
| 1275 | DM-2293 | Internet2 TIER investigation | NULL | 1 |
| 1276 | DM-2294 | Unable to start cmsd on Qserv worker node | Some build issues have qlready been fixed in commit: 9dd378829e8751a6852356967411c20580e2a1c3 Here's the log: {code:bash} [fjammes@ccqserv101 ~]$ cat /qserv/qserv-run/var/log/worker/cmsd.log 150309 21:19:46 9794 Starting on Linux 3.10.0-123.8.1.el7.x86_64 Copr. 2004-2012 Stanford University, xrd version v20140617-203cf45 ++++++ cmsd worker@ccqserv101.in2p3.fr initialization started. Config using configuration file /qserv/qserv-run/etc/lsp.cf =====> all.adminpath /qserv/qserv-run/tmp =====> xrd.port 1094 =====> xrd.network nodnr Config maximum number of connections restricted to 4096 Config maximum number of threads restricted to 2048 Copr. 2007 Stanford University/SLAC cmsd. ++++++ worker@ccqserv101.in2p3.fr phase 1 initialization started. =====> all.role server =====> ofs.osslib libxrdoss.so =====> oss.localroot /qserv/qserv-run/xrootd-run =====> cms.space linger 0 recalc 15 min 10m 11m =====> all.pidpath /qserv/qserv-run/var/run =====> all.adminpath /qserv/qserv-run/tmp =====> all.manager ccqserv100.in2p3.fr:2131 =====> all.export / nolock The following paths are available to the redirector: w / ------ worker@ccqserv101.in2p3.fr phase 1 server initialization completed. ++++++ worker@ccqserv101.in2p3.fr phase 2 server initialization started. Plugin Unable to find required version information for XrdOssGetStorageSystem in osslib libxrdoss.so ------ worker@ccqserv101.in2p3.fr phase 2 server initialization failed. 150309 21:19:46 9794 XrdProtocol: Protocol cmsd could not be loaded ------ cmsd worker@ccqserv101.in2p3.fr:1094 initialization failed {code} | 2 |
| 1277 | DM-2295 | Read through Don's SCADA notes and comment | NULL | 1 |
| 1278 | DM-2299 | Revisit db and dbPool, separate connection from utilities | Revisit whether we need something better than a very basic db connection pool. It may be worth looking at http://docs.sqlalchemy.org/en/rel_0_9/core/pooling.html (or even sqlalchemy in general). Note the Pooling Plain DB-API Connections section - one can use sqlalchemy pooling independently of the other library features. Separate utilities like createDb(), dbExists() and such from core part that deals with connections / sqalchemy. | 12 |
| 1279 | DM-2300 | Improve Webserv/Metaserv | Work includes implementing features requested by SUI (schema metadata, units etc) | 35 |
| 1280 | DM-2301 | Support metadata for databases without description | The metaserv should be able to support databases for which we don't have the ascii schema with descriptions and special tokens (ucd, units etc). This story involves implementing it. In practice, the metaserv/bin/metaBackend will need to be extended to implement "ADD DB" | 4 |
| 1281 | DM-2305 | Measurement transforms for Flux | Provide calibration transforms for flux measurements to magnitudes. | 3 |
| 1282 | DM-2306 | Measurement transforms for centroids | Provide calibration transforms for all algorithms measuring centroids. | 5 |
| 1283 | DM-2307 | Measurement transforms for shapes | Provide calibration transforms for algorithms measuring shapes. | 2 |
| 1284 | DM-2309 | Update dev quick-start guide to new git repositories | The quick-start documentation for developers still points to the old git repositories. The RST document needs to be updated to the GitHub repos. | 1 |
| 1285 | DM-2312 | obs_test's table file is out of date | obs_test's table file is somewhat out of date. Problems include: - afw is required but missing - meas_algorithms and skypix are used by bin/genInputRegistry.py, which is only used to create the input repo so these can be optional - daf_persistence is not used - daf_base is only used by bin/genInputRegistry.py, so it can be optional (though it is presumably setup by daf_butlerUtils in any case) | 1 |
| 1286 | DM-2314 | Improve xssi API to send a few bytes with the message informing the client that a response is available on the server | This would allow Qserv no to send the first protobuf header as a xrootd in-band message, and save some resources (network and CPU due to xrootd/TCP/IP encapsulation) | 6 |
| 1287 | DM-2316 | Clarify expectations for unauthenticated user data access | h4. Short version: Clarify what existing community practices, notably including VO interfaces, appear to rely on the availability of unauthenticated access to information in astronomical archives. h4. Details: At the February DM All Hands, [~frossie] raised an objection when it was mentioned that there is a presumption that all user access to LSST data through the DM interfaces (as opposed to through EPO) will be authenticated. We don't appear to have ever documented an explicit requirement that all access be authenticated. The basic controlling requirement is OSS-REQ-0176, "The LSST Data Management System shall provide open access to all LSST Level 1 and Level 2 Data Products, as defined in the LSST System Requirements and herein, in accordance with LSSTC Board approved policies. ...", which was a carefully crafted indirection at a time when the policy for non-US/Chile access was still being developed. However, this presumption has been around for a long time. It is inherent to the project policy that access to the non-Alert data will be limited to individuals who are entitled to it. No matter what we think the final policy might be, we do have to design a system that can be consistent with this policy. [~frossie] stated that the astronomical community relies on certain types of data and metadata - she mentioned coverage maps, among others - being available through unauthenticated interfaces. This ticket is to ask her (and others) to collect documentation of those existing practices, so that we can figure out what the expectations may be and how to respond to them in our design. | 2 |
| 1288 | DM-2320 | Remove deprecated merging code: rproc::TableMerger | rproc::TableMerger seems to be replaced with rproc::InfileMerger, so this class could certainly be removed easily. | 2 |
| 1289 | DM-2322 | Revisit exceptions in db module | Revisit db/python/lsst/db/exception.py. Perhaps get rid of the numbers. | 5 |
| 1290 | DM-2323 | KT reading list for operational requirements | NULL | 2 |
| 1291 | DM-2324 | Observatory site requirements reading | NULL | 2 |
| 1292 | DM-2327 | Setup hosts for SUI (2x Tomcat, Apache, and build) | Xiuqin's 'short term' version: 1 VM - SUI build server 4GB memory and 200GB hard disk should be good enough. 1 VM - Apache server as a proxy and web front end 4GB memory and 100GB hard disk should be enough port 80 accessible from outside 2 VMs - Tomcat servers each has 16GB memory, access to 1TB of shared hard disk Port 8080 should be open for Apache server to access Port 8009 should be open to each other so they can replicate cache. (First 2 VMs are not absolutely needed. We can always use one of the hosts in number 3 to do build and host Apache server.) The 2 Tomcat servers are larger than we can currently support as VMs. We've discussed repurposing 2 of the older LSST "cluster/condor" nodes (e.g. lsst14 & lsst15) for this purpose. But, ideally these could be implemented with the new vSphere hardware if the timeframe works. | 4 |
| 1293 | DM-2329 | review ""data center in a box" mali. Recover consultant's contact into | reviewed the data center in a box, recovered consultant's name prior to drafting a SOW. | 1 |
| 1294 | DM-2330 | attend DDN WOS briefing, write summary note. | as described above. Summary note is attached. also looked for use of this product in DOE labs, who would be consumers of LSST data. Discovered that it had been investigated for use in HEP a few years earlier, but that is was not adopted because, at that time the hardware and software were coupled. | 1 |
| 1295 | DM-2331 | misc for week of march 9 | finalize job descriptions. Meet with kantor additional hour of orientation for the ISO. group meeting Misc. | 1 |
| 1296 | DM-2334 | Simplify interactions with XrdOss | The qserv code is still using the old ssi scheme for the cmsd, this needs to be rewritten. For details, see https://listserv.slac.stanford.edu/cgi-bin/wa?A1=ind1503&L=QSERV-L#3 | 5 |
| 1297 | DM-2335 | Setup IRODS zone on ISL OpenStack | We begin an examination of iRODS for managing data collections. We perform initial testing using resources available on NCSA's ISL OpenStack. To mock up a zone or 'data grid' managed by iRODS, we set up an ICAT server, a resource server (this is a data storage resource that does not run the central database), and a client host. | 4 |
| 1298 | DM-2336 | Save iRODS installations/servers as docker images | We install and configure iRODS servers (an ICAT server, a resource server, a client host) in docker and make images, pushing the results to a docker hub repository. | 4 |
| 1299 | DM-2337 | develop/propose storage policies | NULL | 6 |
| 1300 | DM-2338 | develop/propose storage procedures | NULL | 10 |
| 1301 | DM-2339 | develop/propose storage implementation | NULL | 3 |
| 1302 | DM-2340 | Reprise SDRP processing metrics | In support of an SDRP-based science talk of Yusra AlSayyad, we spent some cycles gathering/summarizing processing middleware results and metrics from the US side of processing of the Split DRP. This information from notes, logs, databases, etc provided contextual information on the processing campaign that produced the SDRP science results. | 2 |
| 1303 | DM-2341 | Use parallel ssh to manage Qserv on IN2P3 cluster | IN2P3 sysadmin won't manage Qserv through puppet. So Qserv team has to provide ssh scripts to do this. | 5 |
| 1304 | DM-2342 | Integrate changes from Events code review | NULL | 6 |
| 1305 | DM-2343 | Move afw_extensions_rgb functionality into afw proper | See RFC-32 | 1 |
| 1306 | DM-2347 | (In)equality semantics of Coords are confusing | Viz: {code} In [1]: from lsst.afw.coord import Coord In [2]: c1 = Coord("11:11:11", "22:22:22") In [3]: c1 == c1, c1 != c1 Out[3]: (True, False) In [4]: c2 = Coord("33:33:33", "44:44:44") In [5]: c1 == c2, c1 != c2 Out[5]: (False, True) In [6]: c3 = Coord("11:11:11", "22:22:22") In [7]: c1 == c3, c1 != c3 Out[7]: (True, True) {code} {{c1}} is simultaneously equal to *and* not equal to {{c3}}! | 1 |
| 1307 | DM-2348 | useValueEquality and usePointerEquality fail to fail | These SWIG macros return a class instead of raising an exception instance when the equality operation fails. | 1 |
| 1308 | DM-2349 | Add unit tests to SchemaToMeta | Add unit tests, also improve variable names as suggested by K-T in comments in DM-2139 | 1 |
| 1309 | DM-2352 | Install and learn to use iPython notebook | NULL | 4 |
| 1310 | DM-2353 | Participate in design discussion | Participate in the design discussions three times weekly for two to three months. | 9 |
| 1311 | DM-2354 | Participate in design discussion | Participate in the design discussions three times weekly for two to three months. | 9 |
| 1312 | DM-2355 | Participate in design discussion | Participate in the design discussions three times weekly for two to three months. | 9 |
| 1313 | DM-2356 | Identify the hardware resources needed at NCSA for short term development | Supply the hardware resources needed at NCSA for short term development. It is captured in DM-2327 | 1 |
| 1314 | DM-2357 | make PixelFlagsAlgorithm fully configurable | PixelFlagsAlgorithm currently hard-codes the mask planes it considers. This should be fully configurable instead. It also overloads the "edge" flag to mean both "EDGE mask plane was set" and "centroid was off the edge of the image". These should be different flags. We may also want to have this algorithm use SafeCentroidExtractor.' Finally, the algorithm is woefully undertested. | 2 |
| 1315 | DM-2358 | standardize handling of missing peaks in centroiders | GaussianCentroid has a NO_PEAK flag that it sets when there is no Peak to use as an input. SdssCentroid does not. This behavior should be standardized. Maybe we should use SafeCentroidExtractor here? | 1 |
| 1316 | DM-2363 | RGB code introduces dependency on matplotlib | While the new RGB code looks like it's just calling NumPy, NumPy is actually delegating to matplotlib under the hood when it writes RGB(A) arrays. It also turns out that code is broken in matplotlib prior to 1.3.1 (though that shouldn't be a problem for anyone but those who - like me - are trying to use slightly older system Python packages). I think think this means we should add an optional dependency on matplotlib to the afw table file, and condition the running of the test code on matplotlib's presence (and, ideally, having the right version). I'm happy to do this myself (since I'm probably the only one affected by it right now). | 1 |
| 1317 | DM-2364 | Revisit the choice of using flask | We should quickly revisit if flask is the right choice for us. Related: reportedly, our simple flask-based webserver is using more CPU in an idle state than expected. It might be useful to profile things, and look into that. | 1 |
| 1318 | DM-2367 | run lsstswBuild.sh in a clean sandbox | The "driver" script, lsstswBuild.sh, used by the buildbot slave on lsst-dev to initiate a "CI run" has a number of environment assumptions (binaries in the $PATH, paths to various components, hostnames, etc.). This prevents it from [easily] being invoked on any other host. As lsstswBuild.sh builds a number of packages that are not in the lsst_distrib product, the os level dependencies for these other products need to be determined. In addition, the current version of lsstswBuild.sh and related scripts on lsst-dev are not version controlled. | 8 |
| 1319 | DM-2370 | Move QuerySession::_stmtParallel from query::SelectStmtPtrVector to query::SelectStmtPtr | QuerySession::_stmtParallel is a vector but it seems only it's first element is used, so storing it in a vector doesn't seem necessary. Code can be easily simplified here. This should lead to mode understandable code. QuerySession public members and method comments could also be improved here. | 4 |
| 1320 | DM-2371 | run lsstswBuild.sh under Jenkins on EL6 | * Demonstrate lsstswBuild.sh being invoked by jenkins on EL6 (same OS as lsst-dev). * Experiment with a single build slave attached to a jenkins master * Investigate configuration management of jenkins builds. | 6 |
| 1321 | DM-2373 | Improve logger use in qserv | Qserv logger must be easily configurable. Next technique, based on log4cxx documentation allows to do it easily. Example: In QuerySession.cc, initialize a static logger: {code:c++} namespace lsst { namespace qserv { namespace qproc { LOG_LOGGER QuerySession::_logger = LOG_GET("lsst.qserv.qproc.QuerySession"); {code} then use it in Query session member functions: {code:c++} if (LOG_CHECK_LVL(_logger, LOG_LVL_DEBUG)) { std::ostringstream stream; _showFinal(stream); LOGF(_logger, LOG_LVL_DEBUG, "Query Plugins applied:\n %1%" % stream.str()); } {code} And use log4cxx.property to easily configure, AT RUNTIME, logging for each Qserv module class: {code} # logger for all module will inherit this one log4j.logger.lsst.qserv=ERROR # this could be generalized to all Qserv modules: log4j.logger.lsst.qserv.qproc=INFO # can also be done at the class level for advanced debugging #log4j.logger.lsst.qserv.qproc.QuerySession=DEBUG {code} And then in the log: {code} /home/qserv/qserv-run/2015_03/var/log/qserv-czar.log:0319 17:08:48.786 [0x7f208da93740] DEBUG lsst.qserv.qproc.QuerySession (build/qproc/QuerySession.cc:118) - Query Plugins applied: {code} This proposal is a draft and should be improved before implementing it. | 8 |
| 1322 | DM-2375 | evaluate NCSA proposal to investigate CEPH in the context of NCSA Integrated systems lab | The integrated systems lab (ISL) is the orgianizational vehicle used to investigate pre-production technologies at NCSA. Since We still lack the ability to procure goods, I evaluated and commented on an ISL proposal to investigate the CEPH file system for its properties as an alternative to the LSST baseline file system GPFS. | 1 |
