School/Anul 3/Semestrul 1/Proiect Colectiv/StoryPointEstimator/data/duracloud.csv

1	issuekey	title	description	storypoint
2	DURACLOUD-4	Document logging framework	Store-client: Whenever used, creates a c:/ directory and several directories underneath it to store its log files. Sending logs to STDOUT by default and/or having a documented way to changes this would be good.	1
3	DURACLOUD-10	Allow for simple updating of project version numbers	It should be easy/simple to update the version number of all parts of the baseline for each release.	3
4	DURACLOUD-19	Bulk load: Verify successful DuraCloud ingest of 10TB of BHL content	NULL	16
5	DURACLOUD-21	JPEG2K: Image conversion service	Service wrapper and service to perform conversion of images from TIFF to JPEG2K.	8
6	DURACLOUD-22	JPEG2K: Image server service	Service wrapper and service to serve JPEG2K images.	5
7	DURACLOUD-23	JPEG2K: Image viewer service	Service wrapper and service to display JPEG2K images.	3
8	DURACLOUD-25	Messaging: Add support for all DuraStore methods	Add messaging support for all methods available through the DuraStore REST API. Define an appropriate topic heirarchy. Example Uses: - allows for a greater degree of replication filtering - allows for replication on update/delete - allows for creation of a provenance service	3
9	DURACLOUD-28	Replication: handle replication on update/delete	NULL	3
10	DURACLOUD-29	Replication: bulk from existing store to new/empty store	Handles the replication of content from an existing store/space into a new store/space. This allows for starting up a new replicated store.	7
11	DURACLOUD-38	Security: Add application security (basic auth)	NULL	13
12	DURACLOUD-39	Indexing service for space/content properties/tags (Search)	This includes listening for changes to spaces/content and making the calls to have the information included in the search engine index. Note that there needs to be a search engine started and available for this to work. Prior to working this task, a choice will need to be made as to the indexing tools used. The most likely candidates are AWS CloudSearch (http://aws.amazon.com/cloudsearch) and Apache Solr (http://lucene.apache.org/solr/. Various comparisons (http://harish11g.blogspot.in/2012/12/amazon-cloudsearch-vs-apache-solr.html) can be found to assist in this selection. This task does not include updates to the UI or REST API to support users querying the index.	8
13	DURACLOUD-41	Integrity: check on demand (rely on DuraCloud generated hashes)	Should allow for after-ingest checksum validation. Test case: use to validate a replicated content store.	13
14	DURACLOUD-60	Allow expected checksum value to be included when adding a content item	Update the DuraStore REST API call for storing a content item to allow for including an MD5 checksum value which is compared with the stored file's checksum to ensure that the file was received correctly.	2
15	DURACLOUD-61	Handle files larger than 5GB	Amazon S3 limits files size to 5GB, but many of our customers will want to store files larger than this. DuraStore should facilitate this by handling the chunking of large files into smaller files during transfer. Files will need to be stitched back together on download as well. Ideally this will all be transparent to the user, even as far as multiple files appearing as one in the space listing (with the proper total file size.)	3
16	DURACLOUD-62	Enable Pax-logging	As the use of the OSGi container and PAX-tooling increases, it is important that the container enables the configuration of bundle-level logging. Resources: http://ekkescorner.wordpress.com/2009/09/03/osgi-logging-part-2-logservices-vs-classic-logger/ http://fedora-commons.org/jira/browse/FCREPO-630	2
17	DURACLOUD-63	Production AMI continuous integration testing	A suite of automated tests should be created to run on a schedule that: - start instances of production AMIs - check the tabs/fields/buttons of duradmin - start/verify each service - verify content actions	5
18	DURACLOUD-64	Update the development AMI	Update the tomcat user in the development AMI to allow calling of apt-get via sudo with no password. Steps: - call 'visudo' - add this line to the end of the file: tomcat ALL=NOPASSWD:/usr/bin/apt-get Also, add any necessary SystemProperties for enabling servicesutil to initialize $bundle.home	1
19	DURACLOUD-65	Create utility for durastore & duraservice initialization	NULL	1
20	DURACLOUD-66	Initialize servicesadmin $bundle.home at runtime	Currently $bundle.home is a buildtime configuration item. Since the deployment machine will likely have a different setup than any particular build machine, it would be useful to allow $bundle.home to be set at runtime. Note: this has been avoided in the past by building the deployed projects on the deployment machine. Solutions may potentially be: - initialize from REST call - initialize from ~/file baked into AMI - initialize from System property (most tempting)	1
21	DURACLOUD-68	Integrate j2k viewer with duradmin content browse	Although the j2k viewer is currently enabled, it needs to be tied to the duradmin content browse in order to be useful for jpeg2000 content.	3
22	DURACLOUD-69	Modularize project builds	In an attempt to isolate fundamental aspects of the baseline in terms of building and testing, this task is to create three top-level pom.xml that separate: -storage -services -infrastructure Bamboo builds should be created to run each.	1
23	DURACLOUD-70	Cannot upload empty content	While testing Akubra-DuraCloud (vs duracloud trunk r438), I noticed that anytime I attempt to send an empty stream to ContentStore.addContent(..), it behaves as though the request succeeded (no exception is thrown), but the content item never actually gets created in S3. Here's what I've verified so far: 1) S3 supports uploading empty items (verified via S3Fox) 2) ContentStore supports listing, getting, and deleting empty items (tested after uploading an empty item into my test space using S3Fox) 3) Duradmin reports an error (400) when attempting to upload an empty file, but is able to successfully list, get, and delete empty files.	1
24	DURACLOUD-71	IA Filesystem Checksum Verifier	This utility needs to: - walk a directory of downloaded IA books - generate checksums for each file - read IA-manifest of provided checksums (<book>_files.xml) - compare generated to provided checksums	1
25	DURACLOUD-72	content browser: previous link takes the user to the first page.	NULL	1
26	DURACLOUD-74	Create EMC accounts for each pilot partner	In order to allow our partners to use EMC storage during the pilot we need to: 1. Create an EMC account for each partner 2. Add the EMC credentials to the DuraStore initialization for each partner	1
27	DURACLOUD-75	Large-file chunking in durastore	This feature should accept large streams coming in from the durastore-REST-API and chunk them into duracloud.	3
28	DURACLOUD-76	Large-file stitching in durastore	This feature should accept requests for large streams coming in from the durastore-REST-API and stitch the duracloud-chunks into a single response stream.	3
29	DURACLOUD-80	display thumbnails in content browser	NULL	1
30	DURACLOUD-81	Update Rackspace Java library	A few issues have been resolved in the RackSpace Java library based on submitted support tickets. The most current library needs to be added, and RackspaceStorageProvider needs to be updated to account for the fixes.	1
31	DURACLOUD-82	Large-file stitching as client-side utility	NULL	3
32	DURACLOUD-86	Limit count of space items in space metadata	The current call to get space metadata performs a count of the number of items in a space. For S3, this requires getting a list of the entire set of content and counting the included items. When there is a large number of item in the space, this can take quite a while (around 4 minutes for 250,000 objects). Rather than performing this count every time the metadata is requested for a space, the count should be capped and a value returned which indicates that there is a large number of objects. As an example, if the cap for counting is 1000 items, a value of "1000+" could be returned when the cap is reached. Along with this cap, there needs to be a way to get the actual content count. This would likely work best as a client-side count, since the client would be able to update its display to indicate that the count is proceeding.	1
33	DURACLOUD-91	Out of memory error on uploading large files through duradmin.	try uploading three files > 20 MB - tomcat should throw an OOM error.	2
34	DURACLOUD-93	Convert to use SLF4J consistently for logging	There is currently a mix of log4j, commons-logging, and slf4j in the baseline. The log4j and commons-logging dependencies (including transitive dependencies) need to be pulled out in favor of SLF4J. A good reference point is Chris Wilper's completed task doing the same thing in the Fedora baseline: http://fedora-commons.org/jira/browse/FCREPO-630	3
35	DURACLOUD-94	Conversion service cleanup not always happening	In the conversion service, if the call to add the converted file to storage fails, neither the converted file nor the original are removed from the work directory. This cleanup should occur regardless of whether conversion or upload succeed. Additionally, it would be a good idea to add a retry to the upload call.	1
36	DURACLOUD-95	Service Jars should be kept out of the service work directory	In ServicesUtil, specifically ServiceInstallerImpl.explodeAndInstall(), when a service package Zip is unpacked, any jars which are found to already be deployed are being placed in the service's work directory. Instead the jar count should just be incremented (as is done already) and the jar ignored.	1
37	DURACLOUD-96	Unable to add metadata to a content item when the metadata name includes a space	When attempting to add metadata to a content item, if the name of the metadata includes a space, the add fails. There is no failure message, but the metadata is not added to the content metadata list. This issue does not show up when adding metadata to a space. Update: This issue was resolved for Amazon, but in Rackspace a similar problem still exists. On rackspace, if metadata is added to a content item in which the metadata name includes a space, it appears to work properly, but on refresh, that item will disappear.	2
38	DURACLOUD-97	Updating the MIME type of a content item to an invalid MIME value causes errors	If you update the MIME type of a content item to an invalid value (such as image*jpeg) an exception is thrown: java.lang.IllegalArgumentException: Error parsing media type 'image*jpg'. The item can then not be accessed via the REST API (and consequently the UI). There should be a validation check in DuraStore (and DurAdmin) to make sure that the value provided as a MIME type is acceptable.	1
39	DURACLOUD-98	Progress bar for uploading content items in duradmin	right now, adding a content item	1
40	DURACLOUD-101	Create utility to sync to DuraCloud from the filesystem	This can be used to sync arbitrary file system directories with DuraCloud, storing new and updated files. One use case for this is direct backup of Fedora and DSpace file content to DuraCloud (archive to DuraCloud without needing to talk directly.) See http://fedora-commons.org/confluence/x/LwAIAQ for more detail	16
41	DURACLOUD-102	Add option to set color space in Image Conversion service	NYPL images are in proRGB format and need to be converted to sRGB as part of the process of converting TIFF images to JP2. This task is to include the option to indicate the preferred target colorspace in the call to perform the image conversion. This option will need to be included as a config item when deploying the service.	2
42	DURACLOUD-103	Ability for account-admin to add image/mark-up to duradmin home page	The goal of this feature is to allow for DurAdmin to include the branding of the organization using DuraCloud.	1
43	DURACLOUD-104	Handle access control at the DuraCloud level rather than depending on storage providers	Access control as it exists in version 0.2 of DuraCloud is handled by using the ACL and other access mechanisms of the underlying storage providers to allow or disallow direct access to content items in those provider stores. As it has been decided that DuraCloud should always mediate content access, access control should be managed within DuraCloud. Having an Open space will allow anonymous read access to content through the DuraCloud interfaces, whereas having a Closed space will allow access only to authorized users.	2
44	DURACLOUD-106	Security: Add application support for transport security (ssl/tls)	This issue is to enable the baseline to handle both http and https requests.	8
45	DURACLOUD-107	Security: Update PP AMI	NULL	3
46	DURACLOUD-111	Video Streaming Service - Amazon Impl	This service should expose video content through Amazon's Cloud-front streaming server.	8
47	DURACLOUD-114	Multi-thread the Image Conversion Service	Update the Image Conversion Service to be able to spin up a set of threads to handle the conversion process.	2
48	DURACLOUD-115	Image Conversion Service should write output more frequently	The process for reporting image conversion results should be updated so that the output file is created and continually updated as the conversion activity progresses. This makes it is possible to review the results even while the service is running and makes results available even if the service does not complete properly.	2
49	DURACLOUD-118	Stylize 'logout' button	With the addition of security on duradmin, a logout button has been added to the base-header.jsp. It needs to be properly formatted/stylized.	1
50	DURACLOUD-121	Create view link for items when jp2000 service is not running.	NULL	1
51	DURACLOUD-122	Display Generic filetype icon in contents view when jpg2000 service not available.	NULL	2
52	DURACLOUD-123	Implement Browser side caching for space and item metadata	NULL	3
53	DURACLOUD-126	Create content retrieval tool	This tool is the complement to the Sync Tool: A command line utility which would be used to pull content from a DuraCloud space to the local file system. Part of that activity is reconstituting any chunked files.	4
54	DURACLOUD-127	Duraservice REST-API: shorten /duraservice/services to /duraservice	NULL	1
55	DURACLOUD-128	Anonymous StoreClient	The ContentStoreManagerImpl needs to be updated to allow anonymous users to have read access to content. notes from skype: [12:26:26] Bill Branan: the primary store is queried so that a ContentStoreImpl can be created with a StorageProviderType and a storeId. we could pass in StorageProviderType.UNKNOWN and storeId as null. the addStoreIdQueryParameter() method would need to check for null before adding the query param. if it's null, the param is just not included, and durastore reads that as "use the primary store" [12:27:42] Andrew Woods: where is your entry point? [12:28:47] Bill Branan: a new public method on ContentStoreManagerImpl, getContentStoreAsAnonymous() or something	1
56	DURACLOUD-129	Add refresh button to flush clientside cache	NULL	1
57	DURACLOUD-130	Investigate sojo dependency	Hello Danny, I have noticed there have been a couple of build failures due to a new dependency of duradmin that apparently is not always available from its hosting maven repository, namely, net.sf.sojo:sojo-optional:jar:0.5.0. It looks like this is a transitive dependency that was pulled in recently through: net.sf.spring-json:spring-json:jar:1.3.0. Would you be able to do some testing to see if duradmin actually needs this dependency or not. If we do need it, then we can pull it into our own Duraspace maven repository. Since it probably is a runtime dependency with the AJAX tooling, could you remove it from your local maven repository, place an exclude element in your duradmin pom.xml, and run through the application to see if it is actually being used anywhere? ...and naturally double check you local repository to make sure it did not slip back in. Thanks, Andrew p.s. I am planning on putting up a new page/form in duradmin that allows for showing/adding/removing/updating security users. Once the initial page is in the baseline, could you go through and make it usable and presentable? We can talk at length later.	1
58	DURACLOUD-131	Update EMC libraries and re-introduce to baseline	The EMC storage provider is currently excluded from tests and use because of very inconsistent performance around the time that EMC was in the process of taking their storage service into production. They have, hopefully, addressed these problems. The client libraries for EMC need to be updated to the most recent version, our code needs to be updated to work with the latest libraries, and our the tests and use of the EMC provider needs to be re-enabled.	2
59	DURACLOUD-132	Application Initialization tool	As the initialization process for DuraCloud becomes increasingly involved, tooling support would be helpful. This tool should: - read all configuration from a properties files for durastore, duraservice, duradmin, and security - initialize each of the three webapps - set security users on each of the three webapps	3
60	DURACLOUD-133	REST-ful initialization of duradmin	Currently the only way to initialize duradmin is through the ui: duradmin/init.htm This improvement should allow initialization through an http call.	1
61	DURACLOUD-134	Propagate duradmin security updates to durastore & duraservice	When updates to security users happens on the duradmin ui, those changes need to be pushed to durastore and duraservice.	1
62	DURACLOUD-135	Handle XML in a consistent way throughout the DuraCloud baseline	DuraCloud currently uses XML in many places and there are serveral tools and methods used for validating and parsing the XML into an object model. The first step is to pull each bit of XML generation/parsing/validating code into its own class, passing back objects that can be used by the other parts of the code. This will make it easier to re-factor the XML handling into a standard method. We should consider which tool(s) provides the best set of features and use it consistently. We should also have a standard method for validation, Relax NG should be considered here.	5
63	DURACLOUD-140	Error viewing files in DurAdmin space listing	To replicate: Add the attached file (which includes a + in the file name) to a space. View the space using DurAdmin. The request appears to fail to get metadata for the new file, so its link is removed and replaced with: Unable to complete request: status (200) If you click on the link before it disappears, everything appears as it should, the metadata is displayed, and the file can be viewed or downloaded. Note that this problem appears to occur when any of these characters show up in the contentID: +,", #, %, & So a couple of questions here: - What's causing the failure when retrieving metadata? - Why does the link to the item disappear when the metadata can't be retrieved? As a side note, if you select a content item to view on the space page before the metadata for all content items has loaded, all content items in the list turn into error messages. The content page loads shortly after this, so things get back to normal, but it may appear to a user that they have done something wrong by clicking on a content link.	2
64	DURACLOUD-143	Be able to embed J2K viewer in local hosted application	Users would like to view their JPEG2000 images in J2K image server, wrapped in their own UI/portal. Images and image server would be hosted from DuraCloud.	1
65	DURACLOUD-144	Provide users of the Image Conversion service with feedback/status regarding the time that will be required to complete a conversion task	NULL	5
66	DURACLOUD-145	Duraservice: messaging	This feature is analogous DURACLOUD-25, except it calls for applying publishing of messaging events across duraservice methods.	4
67	DURACLOUD-149	ServicesAdmin: Clear registered services	This improvement should provide a RESTful means of clearing the "registered services" from servicesadmin's cache. This would facilitate bringing duraservice and servicesadmin back into sync.	3
68	DURACLOUD-150	DurAdmin: Warning on overwrite	When using DurAdmin, users should be notified when adding a content item will overwrite an existing file.	1
69	DURACLOUD-152	Disable unused projects	This issue should disable /duradavwebapp from the build and remove /sunstorageprovider	1
70	DURACLOUD-153	UI Facelift	NULL	5
71	DURACLOUD-154	Release AMI	This tracker item is a container for all items that should be included in the 0.4 AMI. * redirect / to /duradmin	3
72	DURACLOUD-155	Project Version in batch scripts	This issue is a follow-on to DURCLOUD-10 which enabled true project version numbers in the DuraCloud baseline. This issue is to enable those version numbers to be "sourced" from a common .bat file (ala /resources/scripts/osgi/env.sh & /services/j2kservice/run-deps.sh)	1
73	DURACLOUD-156	License Boiler Plates	Before going open-source. the baseline needs to be updated to include the proper licensing info.	1
74	DURACLOUD-157	Refactor: StorageProviderFactory & TaskProviderFactory	In order to be able to leverage Spring for it IoC, AOP, etc, across these classes (and a chain of classes below them that follow a similar pattern) as well as to more easily unit test them by injecting mocks, this jira-item is to refactor these classes from being comprised of static methods and then use the Spring framework to inject them into their respective *Rest.java counterparts.	1
75	DURACLOUD-158	Change Sync Tool default port	The default port for the Sync Tool is 8080, it should be 443 to allow it to be used by customers without having to set the port to 80 or 443.	1
76	DURACLOUD-159	Provide a way to make use of Amazon's Reduced Redundancy Storage	Amazon's Reduced Redundancy Storage provides a lower cost cloud storage solution. We should add a task which will set content in a space to use this type of storage (and get it back to standard storage.) We could potentially also add a service to allow users to do this directly, but that should be in another tracker item.	2
77	DURACLOUD-160	Create file-based credential store for unit tests	Replace the current unit-test-db with a file-based method for storing and retrieving storage provider credentials. This new method will need to be able to handle storage files which are and which are not encrypted. - Developers should be able to keep a local file (perhaps pointed to by maven settings) which includes all of the account credentials needed for testing. They should be able to choose to encrypt this file or not. Working with an unencrypted file is simpler for local development. Using an encrypted file will allow for more secure storage and for simpler sharing among developers. (The key needed to enc/dec could be stored in maven settings as it is now for the db.) - We may choose to pass an encryped credentials file into bamboo in order to be able to update the test accounts (and keys) without needing to rebuild the AMI. It seems likely that we can reuse the code in app-config to actually read the credentials file, assuming the same format as an app-config file. We should also be able to use the EncryptionUtil in common to handle the encryption/decryption (in combination with Commons-IO FileUtils.readFileToString()). There should be an assembly which provides an easy CLI to encrypt and decrypt a file.	3
78	DURACLOUD-161	Transition to using S3 Reduced Redundancy Storage for all pilot S3 storage	Transition all current S3 objects to use RRS. Update the S3StorageProvider to use RRS as the default. One way to do this would be to add a task which sets a flag within the S3StorageProvider indicating the preferred storage setting for S3.	3
79	DURACLOUD-163	Add config element for duraservice service-registry store credentials	This item should add a new config element to duraservice for the username/password needed to get services from the dcprod service-registry store. Currently these credentials are coming from the hijacked config elements: duraservice.service-compute.username duraservice.service-compute.password	1
80	DURACLOUD-164	500 response from durastore: Too many open files	After uploading some 1-GB files to the WGBH release-0.3 instance, durastore would only respond with a 500 exception. The tomcat log files (attached) indicate "java.net.SocketException: Too many open files" The lsof.out shows an excess of open socket connections to the instance itself. It may be related to the jets3t library (?).	3
81	DURACLOUD-167	Error retrieving content stores	I'm getting this error when trying to access my spaces through the duradmin (accessing files through the durastore appears unaffected) An unexpected error occurred... Error retrieving content stores. Response code was 401, expected value was 200. Response Body: <html><head><title>Apache Tomcat/6.0.18 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 401 - Bad credentials</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>Bad credentials</u></p><p><b>description</b> <u>This request requires HTTP authentication (Bad credentials).</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.18</h3></body></html>Error retrieving content stores. Response code was 401, expected value was 200. Response Body: <html><head><title>Apache Tomcat/6.0.18 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 401 - Bad credentials</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>Bad credentials</u></p><p><b>description</b> <u>This request requires HTTP authentication (Bad credentials).</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.18</h3></body></html>	1
82	DURACLOUD-168	sync tool writes log files into working directory	The sync tools writes a duracloud-common.log file into the current working directory (possibly only an issue on non-windows platforms), writing to the relative file path "./c:/work/programs/duracloud/logs/duracloud-common.log", which causes some errors in the log files: Caused by: org.duracloud.error.ContentStoreException: Error attempting to add content 'c:/work/programs/duracloud/logs/duracloud-common.log' in 'aapp-preservation' due to: Response code was 500, expected value was 201. Response body value: Error attempting to add content 'c:/work/programs/duracloud/logs/duracloud-common.log' in 'aapp-preservation' due to: Content c:/work/programs/duracloud/logs/duracloud-common.log was added to space aapp-preservation but the checksum, either provided or computed enroute, (c2e207b0a44d27e4337d16f93970f1e4) does not match the checksum computed by the storage provider (4f074bb6537ddb42a54f33b401bbd146). This content should be checked or retransmitted. org.duracloud.client.ContentStoreImpl.addContent(ContentStoreImpl.java:394) org.duracloud.chunk.writer.DuracloudContentWriter.addContent(DuracloudContentWriter.java:182) ... 11 more Caused by: org.duracloud.error.ContentStoreException: Response code was 500, expected value was 201. Response body value: Error attempting to add content 'c:/work/programs/duracloud/logs/duracloud-common.log' in 'aapp-preservation' due to: Content c:/work/programs/duracloud/logs/duracloud-common.log was added to space aapp-preservation but the checksum, either provided or computed enroute, (c2e207b0a44d27e4337d16f93970f1e4) does not match the checksum computed by the storage provider (4f074bb6537ddb42a54f33b401bbd146). This content should be checked or retransmitted. org.duracloud.client.ContentStoreImpl.checkResponse(ContentStoreImpl.java:512) org.duracloud.client.ContentStoreImpl.addContent(ContentStoreImpl.java:380) ... 12 more INFO 2010/06/24 16:43:01 [pool-2-thread-3] (SyncWorker.java:61) [retryOnFailure()] - Adding /Volumes/MLAMellonDigLib4/barcode101640/./c:/work/programs/duracloud/logs/duracloud-common.log back to the changed list, another attempt will be made to sync file. ==== md-20013004:barcode101640 chris_beer$ java -jar ~/Downloads/synctool-0.3-driver.jar -d . -f 2000 -h wgbh.duracloud.org -p 443 -s aapp-preservation -t 4 -u admin -w changeme -b .. Starting up the Sync Tool ..................... Startup Complete -------------------------------------- Sync Tool Configuration -------------------------------------- Sync Directories: /Volumes/MLAMellonDigLib4/barcode101640/. DuraStore Host: wgbh.duracloud.org DuraStore Port: 443 DuraStore Username: admin DuraCloud Space ID: aapp-preservation SyncTool Backup Directory: .. SyncTool Poll Frequency: 2000 SyncTool Threads: 4 SyncTool Max File Size: 1073741824 bytes SyncTool Syncing Deletes: false -------------------------------------- -------------------------------------- Sync Tool Help -------------------------------------- The following commands are available: x - Exits the Sync Tool c - Prints the Sync Tool configuration s - Prints the Sync Tool status l <Level> - Changes the log level to <Level> (may be any of DEBUG, INFO, WARN, ERROR) Location of logs: /Volumes/MLAMellonDigLib4/barcode101640/../logs -------------------------------------- s -------------------------------------- Sync Tool Status -------------------------------------- Start Time: 2010-06-24 16:23:07.929 Sync Queue Size: 1 Syncs In Process: 2 Successful Syncs: 30 Failed Syncs: 3 /Volumes/MLAMellonDigLib4/barcode101640/./c:/work/programs/duracloud/logs/duracloud-common.log /Volumes/MLAMellonDigLib4/barcode101640/./c:/work/programs/duracloud/logs/duracloud-common.log /Volumes/MLAMellonDigLib4/barcode101640/./c:/work/programs/duracloud/logs/duracloud-common.log -------------------------------------- quit md-20013004:barcode101640 chris_beer$ tail c\:/work/programs/duracloud/logs/duracloud-common.log /logs/duracloud-common.log back to the changed list, another attempt will be made to sync file. Caused by: org.duracloud.error.ContentStoreException: Error attempting to add content 'c:/work/programs/duracloud/logs/duracloud-common.log' in 'aapp-preservation' due to: Response code was 500, expected value was 201. Response body value: Error attempting to add content 'c:/work/programs/duracloud/logs/duracloud-common.log' in 'aapp-preservation' due to: Content c:/work/programs/duracloud/logs/duracloud-common.log was added to space aapp-preservation but the checksum, either provided or computed enroute, (c2e207b0a44d27e4337d16f93970f1e4) does not match the checksum computed by the storage provider (4f074bb6537ddb42a54f33b401bbd146). This content should be checked or retransmitted. org.duracloud.client.ContentStoreImpl.addContent(ContentStoreImpl.java:394) org.duracloud.chunk.writer.DuracloudContentWriter.addContent(DuracloudContentWriter.java:182) ... 11 more Caused by: org.duracloud.error.ContentStoreException: Response code was 500, expected value was 201. Response body value: Error attempting to add content 'c:/work/programs/duracloud/logs/duracloud-common.log' in 'aapp-preservation' due to: Content c:/work/programs/duracloud/logs/duracloud-common.log was added to space aapp-preservation but the checksum, either provided or computed enroute, (c2e207b0a44d27e4337d16f93970f1e4) does not match the checksum computed by the storage provider (4f074bb6537ddb42a54f33b401bbd146). This content should be checked or retransmitted. org.duracloud.client.ContentStoreImpl.checkResponse(ContentStoreImpl.java:512) org.duracloud.client.ContentStoreImpl.addContent(ContentStoreImpl.java:380) ... 12 more INFO 2010/06/24 16:43:01 [pool-2-thread-3] (SyncWorker.java:61) [retryOnFailure()] - Adding /Volumes/MLAMellonDigLib4/barcode101640/./c:/work/programs/duracloud/logs/duracloud-common.log back to the changed list, another attempt will be made to sync file.	1
83	DURACLOUD-169	non-interactive content transfer tool	as a complement to the sync tool, i'd love to see a non-interactive, one-off content transfer tool (akin to `scp` or `rsync`, either per-directory or per-file) that could be easily integrated into existing workflows, e.g.: $ java -jar duracloud-transfer.jar --file $i -h wgbh.duracloud.org -p 443 -s aapp-preservation -u admin -w changeme ==> transfers file $i into the aapp-preservation space	1
84	DURACLOUD-174	errors loading duradmin spaces	After syncing 500+ large files, I'm getting some weird intermittent errors in the duradmin when trying to verify the files are there.. The answer is probably "don't do that", but reporting it anyway.. For the url https://wgbh.duracloud.org/duradmin/contents.htm?spaceId=aapp-preservation 1) see attached screenshot 2) after refreshing from (1), I the attached stack trace 3) eventually, I get this application error: org.duracloud.duradmin.util.DataRetrievalException: org.duracloud.error.ContentStoreException: Error attempting to get space 'aapp-preservation' due to: Too many open files java.lang.RuntimeException: org.duracloud.duradmin.util.DataRetrievalException: org.duracloud.error.ContentStoreException: Error attempting to get space 'aapp-preservation' due to: Too many open files at org.duracloud.duradmin.contentstore.ContentItemList.getSpace(ContentItemList.java:54) at org.duracloud.duradmin.control.ContentsController.handle(ContentsController.java:71) at org.springframework.web.servlet.mvc.AbstractCommandController.handleRequestInternal(AbstractCommandController.java:84) at [...]	1
85	DURACLOUD-175	Break out OSGI tests into a new project	Move all of the existing service tests which spin up an OSGI container into a new project //services/osgi-test. These tests should run as normal when using "mvn clean install", but should be skipped when the -DskipIntTests or -PskipIntTests flags are used.	2
86	DURACLOUD-177	DuraStore REST-API addition: spaceExists & contentExists	This issue is to add two additional methods to the durastore REST-api: - spaceExists(spaceId) - contentExists(spaceId, contentId)	2
87	DURACLOUD-178	Trade out jets3t libraries under S3StorageProvider for Amazon-supported libraries	Use the Amazon libraries (http://aws.amazon.com/sdkforjava/) for connecting to S3. They will also be used for connecting to EC2.	4
88	DURACLOUD-179	Set higher file handle limits in DuraCloud AMIs	WGBH has seen issues on their instance due to the tomcat process running out of file handles. To get around this the changes listed at the bottom of the AMI creation wiki page (https://wiki.duraspace.org/display/neworg/DuraCloud+AMI+Setup#DuraCloudAMISetup-AdditionalNotes) were added. These changes need to be included in the DuraCloud AMIs.	1
89	DURACLOUD-180	Sync Tool run to completion	Add an option to the Sync Tool which would allow it to run until its work queue is empty, then exit. This would be useful for administrators who would like to use the Sync Tool as part of a cron job to sync all of the day's updates at one time rather than have the Sync Tool run continually.	1
90	DURACLOUD-181	Sync Tool retry failed transfers	Update the Sync Tool to write failed transfers to a file in the backup directory, then add an option to retry those failed transfers on startup. This would need to be a one time retry (otherwise the risk of a loop is too high) which pulls the contents on the failed transfers file into the work queue, then clears that file; any new failures would be written to the file again.	2
91	DURACLOUD-182	Sync Tool start from scratch option	Add an option to the Sync Tool which would clear out its status logs (or just ignore them) in order to start from scratch (attempt to sync all files) rather than restarting. This would remove the need to ever manually clear the backup directory unless wanting to remove log files.	1
92	DURACLOUD-185	change icon to something more appropriate on deploy/undeploy service.	not sure if the lock/unlock icons used for open/closed spaces really applies to service deploy/undeploy (0.6)	1
93	DURACLOUD-187	After logging in, redirect the user to the requested URL	NULL	1
94	DURACLOUD-188	provide next and previous links for navigating content items.	NULL	1
95	DURACLOUD-190	Fix Duradmin UI integration tests.	NULL	1
96	DURACLOUD-191	Update the Image Conversion service to make use of Amazon's Elastic MapReduce service	In order to handle processing the volume of data being presented by our pilot partners, DuraCloud needs to be able to distribute compute work onto multiple machines. It appears that the most standardized way of doing this distribution is to make use of Apache Hadoop. Since Amazon provides the Elastic MapReduce service, which makes use of Hadoop, it makes sense to investigate that as the initial step towards being able to take advantage of scalable processing in the cloud. We already have data to convert (NYPL tiff images) and an existing Image Conversion service, this task is to allow the image conversion to run via MapReduce.	16
97	DURACLOUD-192	Full URL of streaming content	This improvement is to expose the full URL, including the streaming host server, of content that is streamed via the mediastreaming service. Various stream viewers provide the ability to paste in such a URL for testing.	1
98	DURACLOUD-193	Navigating away from an upload in progress causes the upload to fail.	The quick fix is to provide a warning to the user. The better solution is probably to make the spaces,services,admin,and dashboard pages all driven off the same "page" and making the various sections (spaces,services,etc) addressable after the hashmark.	1
99	DURACLOUD-195	The uploads dialog gets a little messy when you have >4 files that have been uploaded (close button sits over other stuff, can't see the bottom of the scroll bar, etc)	# The uploads dialog gets a little messy when you have >4 files that have been uploaded (close button sits over other stuff, can't see the bottom of the scroll bar, etc)	1
100	DURACLOUD-196	Mark image files when the space is closed, indicating that the JP2 viewer is only available for open spaces	It would be really useful to have a notice box on image files when the space is closed, indicating that the JP2 viewer is only available for open spaces (then do viewing as if the JP2 service were not deployed).	1
101	DURACLOUD-197	Make available services more explicit.	on the services tab, you have to click "deploy a new service" to view a list of other available services not currently running. it would be nice if that information was somehow presented on the immediate service page, as it wasn't immediately apparent that there were other services that could be run.	1
102	DURACLOUD-198	DurAdmin: Add Content Item button is available prior to selecting a space	The "Add Content Item" button on the spaces tab of DurAdmin can be selected without choosing a space. The add content dialog box opens and allows you to interact as usual, but when you click the Add button nothing happens. The "Add Content Item" button should likely be grayed out until a space is selected.	1
103	DURACLOUD-200	New StorageProvider: MS-Azure	This feature should create a StorageProvider implementation ala EMCStorageProvider.java RackspaceStorageProvider.java S3StorageProvider.java for MicroSoft's Azure storage offering. http://research.microsoft.com/azure	13
104	DURACLOUD-202	Server Health and Status Monitoring	This is a top-level feature that will span from the UI dashboard to the backend status. Exactly what functionality is expected in this feature needs further investigation. Examples may include: usage - storage requests, bandwidth consumption, uploads, downloads, etc storage breakdowns - by space, mimetype, storage provider, etc cost estimating see the following page for further ideas: https://wiki.duraspace.org/display/DSPINT/DurAdmin+Dashboard	13
105	DURACLOUD-204	Persisting User Passwords	This improvement should persist system and user credentials to enable the same ones to be used across migrations to a new DuraCloud instance.	1
106	DURACLOUD-207	Service configuration: Exclusion groups	The task is to update the service-config schemas to support the notion of exclusion groups during deployment and configuration of services. This allows the UI to dynamically display the appropriate input elements based on initial selections made by the user. See DURACLOUD-215	2
107	DURACLOUD-208	Bulk Delete Space	NULL	1
108	DURACLOUD-209	Bulk Delete Content	NULL	1
109	DURACLOUD-214	Bulk Update Content Mime Type	NULL	1
110	DURACLOUD-215	Service Exclusion Groups - UI Modifications	UI changes to support DURACLOUD-207	1
111	DURACLOUD-217	Chrome & IE mis-behavior	On Wed, Aug 4, 2010 at 12:21 PM, Bryan Beecher <bryan@umich.edu> wrote: OK, this is really weird. Chrome throws a tomcat error. IE hangs. Firefox works A-OK. I'll use Firefox. :-) -- bryan On Wed, Aug 4, 2010 at 12:18 PM, Bryan Beecher <bryan@umich.edu> wrote: Hey, Carissa, I logged in, changed the password, and after hitting submit, got a page with this: {"user":{"accountNonExpired":true,"accountNonLocked":true,"credentialsNonExpired":true,"enabled":true,"grantedAuthorities":["ROLE_ADMIN","ROLE_USER"],"password":"[omitted]","username":"admin"}} which contains the new password. However, whenever I navigate to the home page (URL below), I get a 500 response from tomcat, and a long stack trace (which I can send it if it helpful). If I cut-n-paste the URL into a different browser, I do get the login screen, but it just hangs when I try to authenticate. -- bryan	1
112	DURACLOUD-218	Ability to view the actual number of items in a space when the count exceeds current limit (1000 now)	NULL	1
113	DURACLOUD-219	In Service UI: Config : Exclusions - clarify relationship to a "mode" and the user configs that go with it.	NULL	1
114	DURACLOUD-220	Service Report Dashboard	This task is to create a UI dashboard for displaying and downloading reports on service jobs. Information about both running services and completed services will need to be displayed. REST API calls should be available from which the service report information can be gathered. A UI will need to be created to be able to display the information in the reports. The reports will likely be in XML, so XSLT is an option for display.	5
115	DURACLOUD-221	On session timeout, make sure javascript forces the user to login (instead of failing ungracefully)	NULL	1
116	DURACLOUD-222	Service configuration: schema update for nested modes	This improvement is to update the service-types.xsd to define serviceType as its current sequence of config elements as well as a modeType that itself contains another modeType and the sequence of config elements.	1
117	DURACLOUD-223	Service Report: backend utility for completed services	This feature provides the backend support for providing service summary details for services that have completed. 'Completed' is defined as either coming to a state of 'success' or 'failure', or in the case of services that just stay running (e.g. Image-Server) coming to the state 'started'. Service summaries should be collected as soon as the service completes and be persisted to a single file containing all services. This single report file will start fresh on a monthly basis.	5
118	DURACLOUD-224	DurAdmin - enter key does not properly submit content detail edit	When editing the mime-type of a content item in DurAdmin (using the "edit content item" dialog), using the enter key to submit does not save the value which was input and returns the user to the main spaces view with no space selected. Using the Save button works just fine. Using the enter key should perform the same function as clicking Save.	1
119	DURACLOUD-225	DurAdmin - next and previous buttons for content list are displayed even when not necessary	The recently added next and previous buttons to allow cycling through large sets of content items seem to show up even where are only a few content items in a space. Those buttons should only be available when there are more content items than can be displayed in the scroll window.	1
120	DURACLOUD-226	DurAdmin - "Login" button on login screen does not respond to Enter key on Chrome	When using the Google Chrome browser and attempting to log in via DurAdmin, the Login button is not triggered by hitting the Enter key. Clicking on the Login button works fine. In Firefox and IE both clicking on the button and hitting the Enter key work fine.	1
121	DURACLOUD-228	Cannot display Spaces in DuraAdmin UI	After logging into the DuraAdmin web app I can navigate to most tabs successfully, but when selecting the Spaces tab, I get a pop-up that says only "error:undefined". And the sync tool also exhibits a problem trying to access Spaces: Starting up the Sync Tool ...Exception in thread "main" java.lang.RuntimeException: Could not connect to space with ID 'synctest4'. at org.duracloud.sync.endpoint.DuraStoreSyncEndpoint.ensureSpaceExists(DuraStoreSyncEndpoint.java:82) at org.duracloud.sync.endpoint.DuraStoreSyncEndpoint.<init>(DuraStoreSyncEndpoint.java:69) at org.duracloud.sync.endpoint.DuraStoreChunkSyncEndpoint.<init>(DuraStoreChunkSyncEndpoint.java:33) at org.duracloud.sync.SyncTool.startSyncManager(SyncTool.java:89) at org.duracloud.sync.SyncTool.runSyncTool(SyncTool.java:185) at org.duracloud.sync.SyncTool.main(SyncTool.java:240)	1
122	DURACLOUD-229	Content does not complete if list size >= 1000	When the number of content items within a space exceeds 1000 and the "maxResults" parameter is set to 1000 or greater (1000 is the default), then ContentIterator.java eventually terminates early. This is because the underlying query to the storage provider is maxResults+1 to account for the possibility that the space metadata content object may be in the listing. Since the S3 client will only return a max of 1000 items regardless of the maxResults parameter, the call with 1001 effectively turns into a call for 1000 items. The logic in ContentIterator is terminating if the number of items in the listing returned is less than the maxResults parameter, which is the case when 1000 items are returned and one is removed (i.e. the space metadata object).	1
123	DURACLOUD-230	Exception thrown on duradmin initialization	The attached stacktrace has begun appearing in the catalina.out log when running the app-config initialization tool. By downloading various revisions of the baseline, I was able to associate the origins of the stacktrace to svn r[47], on Aug 2nd. http://groups.google.com/group/duracloud-codewatch/browse_thread/thread/4dcd83462d243272?hl=en# I observe the stacktrace when running mvn clean install -DskipTests from the top of the baseline, followed by java -jar app-config/target/app-config-0.5.0-SNAPSHOT-driver.jar localhost.properties It is unclear to me if the mods to trunk/duradmin/src/main/webapp/WEB-INF/config/tiles.xml trunk/duradmin/src/main/webapp/WEB-INF/jsp/error.jsp cause the error, or just allow it to be visible in the catalina.out log. As a note, I also get the same result when running the following curl command: curl -d duradmin-init.xml -u un:pw http://localhost:8080/duradmin/init	1
124	DURACLOUD-231	Allow users to select S3 storage class	As of release 0.6: - All content added to S3 is set to the reduced redundancy storage class - There are tasks available which will transition all content in a given space to a particular storage class (either standard or reduced redundancy) This task is to: - Provide a way for the user to select which storage class they would prefer to be set when new content is added. This selection would be for all content in an account. Later tasks will need to handle: - Provide selection of storage class at the space level - Provide a way for users to transition their existing content to their preferred storage class. This will likely be a service which leverages the existing tasks.	3
125	DURACLOUD-232	REST-API: Create top-level exception barrier	This improvement is to create a top-level exception barrier for the durastore and duraservice REST-APIs. Currently, unexpected exceptions appear in the client response as either a stacktrace or a tomcat html error document.	1
126	DURACLOUD-233	Add support for CloudFront invalidation	Since the media streaming service was created, Amazon has added a feature called Invalidation, which provides an API call to remove content from CloudFront. The media streaming service should be updated to make use of this call when stopping the service, so content will no longer be available for streaming when the service has been turned off.	1
127	DURACLOUD-235	Metadata name prefix mismatch between durastore & duradmin	The duradmin metadata prefix is an extension of the durastore metadata prefix durastore: x-dura-meta- duradmin: x-dura-meta-ext- Therefore, metadata that is created through the durastore REST-API may not contain the full prefix that duradmin is expecting. These two prefixes need to be made consistent, and the logic should probably be pushed down from duradmin into storeclient.	3
128	DURACLOUD-237	Test Infrastructure: mocking integration tests	This improvement is to allow developers to run the test suites more often by reducing the amount of time it takes to run a build. There are many integration tests that unnecessarily go all the way to the underlying storageproviders. These tests should be revisited with an eye towards mocking out the network calls where they are overly redundant.	8
129	DURACLOUD-238	Test Infrastructure: reinstate bamboo	NULL	1
130	DURACLOUD-239	DurAdmin: Uploaded file can be displayed in the wrong space	When uploading a file, when the upload completes, the content Id of the uploaded file is added to the list of items in the currently viewed space, even if that is not the space where the item has been stored. Clicking on the content Id then brings up a "get item failed: error" dialog. Clicking on the either space (the one being viewed or the one where the upload landed) allows them to refresh and show the correct listing.	1
131	DURACLOUD-240	DurAdmin: Images in closed spaces should be displayed as if no J2K service exists	Re: DURACLOUD-196, when a space is closed, the View link still attempts to open it in a J2K viewer. Along with the current notice (and link to open the space), the images should act like the J2K service is not deployed until the space is opened.	1
132	DURACLOUD-249	Fedora Integration: Basic-Auth for externally referenced datastreams	This feature is the DuraCloud marker for the Fedora issue: https://jira.duraspace.org/browse/FCREPO-748	1
133	DURACLOUD-252	Fixity Service: Amazon Elastic Map Reduce version	NULL	8
134	DURACLOUD-254	Bit Integrity Service: update UI based on new service-config schema	NULL	4
135	DURACLOUD-255	Incorrect redirects from http to https	Saw this while testing with Poster: When performing a PUT to add a new content item, if everything is right, except you use http rather than https in the URL you end up with a 404 response. Using httpfox, I can see that what happens is that a 301 is sent back and the original request is forwarded to the https address, but as a GET rather than a PUT. So the 404 is the result of attempting a GET on a content item which does not exist. It would appear that the HTTP verb is lost in the redirect.	2
136	DURACLOUD-256	On some machines, not able to login to Duradmin	Brian Beecher and Andrew have seen occasional problems logging in. The symptom: user tries logging in with correct credentials - no failure, no success. Brian is using Chrome - doesn't see the problem in "In cognito" mode. Seems to be a caching issue.	1
137	DURACLOUD-257	Use Amazon java client for Cloudfront	DURACLOUD-178 transitioned the majority of the interactions over to using the Amazon-provided java client. Unfortunately, when that work was done, that client did not yet support Cloudfront, so access to Cloudfront continued to be through Jets3t. Once the Amazon library is updated to include Cloudfront, this task is to complete the transition and remove Jets3t from the S3StorageProvider dependencies.	1
138	DURACLOUD-259	Create 0.7 AMI	This production AMI should 1. contain removal of $BUNDLE_HOME/container & /work & /attic on shutdown in the /etc/init.d/duraspace script 2. update /etc/apache2/httpd.conf to include the following line: RewriteLock /var/lock/apache2/apache-rewrite.lock {new} RewriteMap local-ip-map prg:/etc/apache2/bin/local-ip.sh {already present} 3. touch: /var/lock/apache2/apache-rewrite.lock	2
139	DURACLOUD-260	Login failure on some browsers/environments	Firefox 3.6.10 Chrome 6.0.472.62 Opera 10.62-6438 Occasionally on Firefox and Chrome the first time a username/password is provided and the 'enter' key hit, the fields just clear. But immediately re-inserting the values has always succeeded.	1
140	DURACLOUD-261	Allow storeclient to handle redirects	Currently, if the storeclient is initialized to use port 80 to connect to durastore, it will fail when making calls because the responses are 301s which redirect to port 443. This task is to allow the storeclient to retry using the redirect path rather than failing. This update would likely benefit the serviceclient as well.	2
141	DURACLOUD-265	Ensure integrity of input content to hadoop services	This improvement is to revisit the hadoop services: - fixity - bulk image conversion - replication on demand and ensure the content pulled from S3 is valid with respect to its stored MD5 value.	3
142	DURACLOUD-266	Notification on service completion.	This task is to provide a notification to the user running a service when that service completes. The notification should include (a link to) the final output report, if one was generated by the service. Original description: Bryan Beecher suggested that adding some sort of pop-up, email message, or other indicator might be a good idea once the fixity service has finished the job that it is running. It isn't clear how to tell when it is finished beyond polling the Services page.	5
143	DURACLOUD-267	Add a UI to the retrieval utility	NULL	5
144	DURACLOUD-270	UI Pagination after "delete all"	The scenario is revealed when wanting to clear the entire contents of a space or the first few hundred items. 1. have a space with many items (+1000, this is commonly the case with the log output from hadoop jobs) 2. select the space and select the check-box that marks all of the items shown on the current page 3. select "Delete Selected Content Items" 4. notice all items are deleted 5. select the "next" button 6. notice the next page of items is not displayed 7. refresh space (by hitting the space name) and notice the next page of items displays	1
145	DURACLOUD-271	UI Message when creating space that already exists	When creating a space that already exists, the following pop-up displays the error: "error: space is undefined" Normally I can determine that the error must relate to the fact that the space already exists. But sometimes in creating a new space, the action fails and the space is not listed on the left-hand side listing of spaces. But despite the failure, the space is actually created (although not shown). So trying to create it again shows the "error: space is undefined" message which does not make it very easy to determine the actual error. The suggestion is to render a more specific error message.	1
146	DURACLOUD-272	Metadata add button label is removed on focus	When adding metadata to either a space or content, the add metadata button label ("+") is removed as soon as the button gets focus. This follows the [name] and [value] text being removed from the metadata text fields, but it should not happen on the button.	1
147	DURACLOUD-274	Handle filenames which include spaces	This issue is to review the current set of DuraCloud services and utilities (including the sync and retrieval tools) to determine which ones do not currently support working with content items which include a space (" ") character in their content IDs. After the list is identified, each service and utility should be updated to handle this case.	3
148	DURACLOUD-275	DuraAdmin - View images in closed space in the J2K Viewer	Remove the notice that the space needs to be open in order to view the image in the J2K viewer. Change the link to view closed images in J2K viewer. Also change viewer url param from http to https. Make sure to release with issue DURACLOUD-138	1
149	DURACLOUD-276	Update sync tool command line flags to match retrieval tool flags	The following flags need to be updated in the sync tool to match the retrieval tool: 1. password: change to -p 2. port: change to -r 3. backup (work) directory: change to -w 4. sync (content) directory: change to -c 5. sync deletes: change to -d 6. exit on completion: change to -x 7. add: store id: -i	1
150	DURACLOUD-277	Update retrieval tool to skip space dir creation on single space retrieval	It is not necessary for the retrieval tool to create a directory for the space being retrieved when retrieval is only for a single space. This task is to update the retrieval tool such that when only one space is being retrieved, the files go directly into the content directory. When multiple spaces are retrieved, space directories are created, as is the case currently.	1
151	DURACLOUD-278	Selecting a single space for deletion displays "delete multiple spaces?" warning	In DurAdmin, if you select a single space for deletion, on the far right you'll see a "Multiple spaces selected" message. After clicking "Delete Selected Spaces", you'll also get a warning that says "Are you sure you want to delete multiple spaces?". Obviously, this is a little confusing, as it causes you to have to double check that you aren't accidentally about to delete multiple spaces.	1
152	DURACLOUD-279	Bulk Image Transformation failures	The Bulk Image Transformation service has failed in numerous tests which has been over a set of TIFF images files supplied by Rhodes. This set includes 317 images, ranging in size from 2.35 MB to 265.42 MB. The primary test case has been converting to JP2 format using the source color space. The service has failed when running with 1 instance of each instance size. The service has also failed in a test running an xlarge instance with only 2 mappers. Watching the memory consumption on a small instance while the service is running shows that a significant amount of memory is consumed by the mogrify step (which is run using a bash script to call ImageMagick) - all swap space was consumed, and less than 10% of memory remained available. Testing has indicated that the size of images being transformed likely plays a role in the failures. Moving the two largest Rhodes files to another space (all of the rest of the tiffs being under 115 MB) allowed the transformation to complete successfully using 3 xlarge instances. Running the conversion over the same set with only 1 instance (regardless of size) still failed, however. The attached file shows the list of tiff files which were successfully transformed in these tests, as well as the files on which the tests failed. The large instance was run with two mappers, and the x-large instance with 4 mappers.	5
153	DURACLOUD-281	Duraservice: Resolve spaces for all providers	This improvement is to have ServiceManager.java resolve the spaces for all providers when a service config object specifies a "ALL_STORE_SPACES" mode.	1
154	DURACLOUD-282	Bit Integrity Service: update service config for provider spaces	This improvement is to change the Bit Integrity Service to leverage the 'mode' schema element to specify the need for resolving spaces for all providers.	3
155	DURACLOUD-283	Duplicate on Upload: ensure metadata is included	This improvement should ensure that content that is replicated via the "Duplicate on Ingest" service should also duplicate the user-supplied metadata.	1
156	DURACLOUD-284	Integrate Droid for Mimetype detection	NULL	2
157	DURACLOUD-285	Bottleneck in hadoop processing	Two runs of the Bulk Bit Integrity service over the same 1TB set of BHL content took approximately the same amount of time (a few minutes over 8 hours.) The problem is that the first run used 10 small instances and the second run used 10 x-large instances. The x-large instance should have completed the task much faster, not only because they have higher bandwidth and compute capacities, but also because they are set to use 8 mappers per instance rather than the small instance setting of 2. This task is to discover and fix the bottleneck which is not allowing the x-large instances to run at full capacity.	3
158	DURACLOUD-286	Allow system services to be deployed along with services which depend on them	This task will remove the system services (imagemagick, webapp util) from the list of deploy-able services and instead have them be deployed as needed when dependent services are deployed. Likely this will mean bundling the jars for these services into the services for which they are a dependency, and updating them to start when they are deployed into OSGI.	2
159	DURACLOUD-287	Odd characters seemingly replace copyright symbol -- UTF8 issue?	At the bottom of the DurAdmin UI it says, "�2010 DuraSpace.org".	1
160	DURACLOUD-288	MS-AZURE Space item count is off by one	The number of items listed in an empty AZURE space is -1. The number of items listed in an AZURE space with one content item is 0. etc...	1
161	DURACLOUD-289	Retrieval Tool to retrieve list of specific files	I think the retrieval tool is missing an option where you can retrieve specific file(s) from a specific space. So, in the DSpace use case, when doing a partial restore, you may wish to retrieve only a portion of the files from the space (rather than having to pull down all of them). I would like to be able to request specific files by name (or by a comma separated list of names). A regular expression could also work, but a list of exact names would seemingly meet the needs for DSpace. So, for DSpace, the AIP files stored in DuraCloud will have very specific names, similar to: COLLECTION@123456789-10.zip ITEM@123456789-11.zip Generally the AIP name is a specific format: <DSpace-Object-Type>@<DSpace-Object-identifier>.zip So, if a DSpace install wanted to restore just a particular object (or small set of objects), it might be nice if they can request *just* those objects by name (rather than having to pull down their entire space). I just know this is a specific case that's currently missing -- we either need a simple solution from the Retrieval tool, or we need a DSpace-specific tool to support partial content restores (where a user only wants to restore a particular set of items or a single collection).	2
162	DURACLOUD-291	Create 64-bit dev AMI	NULL	2
163	DURACLOUD-293	Improve performance for retrieve content metadata in AzureStorageProvider	Move the generation of content checksum from getContentMetadata to addContent. Also waiting for feedback from the Azure team to find out why they do not provide the md5 in the eTag.	1
164	DURACLOUD-294	Create 0.7.1 release	AMI: ami-24788f4d Docs and artifacts have been created and pushed to dcprod and atlas per: https://wiki.duraspace.org/display/DSPINT/Release-Related+Scripts	1
165	DURACLOUD-295	When using the previous and next buttons in the DuraCloud Administrator interface, the "Space Detail" section on the right disappears.	When navigating through the list of content items for a space using the previous and next buttons in the DuraCloud Administrator interface, the "Space Detail" section on the right disappears and only reappears once you re-select the space or click on an individual item.	1
166	DURACLOUD-298	Bit Integrity Service: split into two services	The clean of the Bit Integrity service should include: - remove 'salt' from UI - split the service into two services: -- Bit Integrity Tools (generate & compare) -- Bit Integrity (verify)	1
167	DURACLOUD-299	Cross-Service Clean up	The clean should include: 1. change order of services to be logical (or possibly alphabetical) 2. change name of services in UI to have the same prefix if they are variations on the same service - example: Bit Integrity Service - Bulk	1
168	DURACLOUD-301	Provide a clear visual cue in DurAdmin to indicate which storage provider is currently selected	It has been observed that, when using DurAdmin, it can be easy to forget which storage provider is selected and perform an action on the wrong provider. This task is to provide a visual cue to help distinguish between providers. One potential way to do this would be for each storage provider to correspond with a different color, and that color be displayed prominently (perhaps even as the spaces area background) when that provider is selected.	4
169	DURACLOUD-302	Remove "Location" selection on service deployment via DurAdmin	Being able to select an alternate deployment location when deploying a service is a feature to come down the road, but in the meantime, the UI should not display this option when none of the services use it.	1
170	DURACLOUD-303	Simplify deployment of bulk services	Items to simplify for bulk services (Duplicate on Demand, Bulk Image Conversion, Bulk Bit Integrity Checker): 1. Use a default setting for number and type of instances. The number and type will depend on the service and will require testing to determine a reasonable default. (10 large instances should work for Bulk Bit Integrity, testing by the pilot partners should help narrow in on good defaults for Duplicate on Demand and Bulk Image Conversion)	3
171	DURACLOUD-304	Update Image Transformer service deployment options	These changes apply to both the Image Transformer and Bulk Image Transformer services: - Clarify the "Destination Color Space" such that it is not confused with a DuraCloud space - Move file name prefix/suffix options directly under source space	1
172	DURACLOUD-305	Azure Storage Provider - list space contents contains truncated contentId	The name attribute returned from IBlobContainer.listBlobs sometimes truncates the contentId. Instead parse the contentId from the uri attribute until Azure4J fixes this issue - https://sourceforge.net/tracker/?func=detail&aid=3116592&group_id=266876&atid=1135911 http://www.windowsazure4j.org/learn/java_docs/org/soyatec/windowsazure/blob/IBlobProperties.html	3
173	DURACLOUD-306	Bit Integrity Service: simplify user options	This improvement should update the Bit Integrity Service in the following ways: - create a default space for output - create auto-generated output name - provide checkbox for providing input listing name - add a checkbox for "I do not have an input listing", which cause an input listing to be generated from metadata	3
174	DURACLOUD-307	Bulk Bit Integrity Service: simplify user options	This improvement should include the following updates to the Bulk Bit Integrity Service: - remove config option from ui/xml: "destination space" - have service output automatically right to the space "x-service-out", with the content prefix "bitintegrity-bulk/" - remove config option from ui/xml: "working space" - have service automatically set working space to "x-service-work"	2
175	DURACLOUD-309	Audit log service	This task is to capture and make available to users audit logs which list the activities that have occurred in DuraCloud. There will likely need to be a separate set of logs for storage and services. Once messaging support has been completed on storage and services, a new service could be created to listen to the messages being sent and store write those to log files which are stored in the user's storage area.	3
176	DURACLOUD-310	Audit logging: Storage	This task is to capture all of the audit information relating to storage events for all users in a single location. This listener will be running at all times for all users and storing the results within DuraCloud storage.	5
177	DURACLOUD-311	sync tool syncing .ds_store files from mac	The sync tool syncs .ds_store files and other peripheral files that are autogenerated by a mac in a folder but that are not actually content that needs to be synced to DuraCloud. Would be nice if the tool knew to automatically ignore these types of files.	2
178	DURACLOUD-312	ServiceInfo.clone()	The clone() method needs to be implemented in order for "deployedServices" within ServiceManager.java not to have their SystemConfig data member become null after deployment when it is currently going out of scope.	1
179	DURACLOUD-313	Security initialization	This improvement should disallow updates to the root user security credentials via the REST API.	2
180	DURACLOUD-314	Bulk Bit Integrity Service: error allocating memory	This issue is to resolve the bug recorded at the following link: https://wiki.duraspace.org/display/duracloudpilot/Reported+Issues From Bryan Beecher (ICPSR) on 12/3 The hadoop logs are attached. A brief scan of the logs indicates that the hadoop servers are running out of memory.	3
181	DURACLOUD-315	Storage Usage Report Service	Provide a way to generate a report with at least the following information: - Number of files for each space - Total size of files stored in each space - Average file size for each space - File types (mimetypes) represented in each space - Total number of files (all spaces) - Total size of all files (all spaces) - Average size of file (all spaces) - File types (mimetypes) represented (all spaces) - All of the above information for each storage provider in use These reports should be available for download and should be stored within DuraCloud so that they can be referenced periodically. At some point we may want to provide a way to schedule the running of such reports (monthly). This would seem to be a good fit for a service which gathers the needed information, formats it, and stores it in an output space. A REST-api should be exposed for client access to the reports. This feature was suggested by Markus Wust (NCSU) during a pilot partner meeting on Dec 13, 2010. He was also interested in seeing how the information included in these reports relates to storage costs.	13
182	DURACLOUD-318	Bit Integrity Service: Incorrect report when item on list does not exist	From David Arjanik (Columbia) on 12/15 Was unable to successfully complete Use Case 4: Preservation Monitoring Test. Here's an excerpt from my notes regarding 2.c of the use case test plan: c. Service Test 1 i. Using the initial listing from Test0 above, create a new listing file with some MD5s altered some entire line items removed some new line items Arjanik: Done. Altered checksums of first three lines, deleted the next three lines, and added two lines at the end for files not in the DuraCloud instance. Uploaded to "admin" as usecase4-service-test-1.md5 ii. Run fixity service with this altered listing as input as in the previous test Arjanik: Reconfigured service to run against this new md5 file. Report ran successfully, generating files fingerprints-service-test-1.csv and integrity-report-service-test-1.csv (names I specified in the reconfiguration). iii. Verify that erroneous entries were identified in the generated report Arjanik: Downloaded fingerprints-service-test-1.csv and integrity-report-service-test-1.csv. Arjanik: Checked integrity-report-service-test-1.csv. This file is attached. Only one line of report data: | space-id | content-id | 0:admin/usecase4-service-test-1.md5 | 1:admin/fingerprints-service-test-1.csv | status | | Invalid manifest file: newspace | README.txt | Error: metadata is null | for: newspace/README.txt | Arjanik: Deleted the new md5 file and the generated report files for this service test from the "admin" space to try again. Reconfigured and re-deployed the service. Downloaded the results. Same results. Edited the md5 file and replaced fictitious space "newspace" with "usecase3". Deleted the old report files and re-deployed the service. Downloaded the reports. Renamed the integrity report to integrity-report-service-test-1-2nd-try.csv and attached. Results are basically the same as before. Opened the fingerprints file. The new lines were correctly identified as referring to files that didn't exist in DuraCloud. The six files that I've attached: PERFECT MATCH TEST usecase4.md5 - this is the input file listing correct space contents and correct md5 checksums fingerprints.csv - this is the md5 checksum file generated by the Bit Integrity Checker integrity-report.csv - this is the diff report generated by Bit Integrity Checker This test was successful. ERRORS TEST: - The checksums of the first three files in the list were changed - The next three files were removed from the list. - The last two lines refer to files that are not in our DuraCloud instance. fingerprints-12-15-2010.csv - this file appears to use incorrect formatting in the last two lines, the lines with files that don't exist in the instance. integrity-report-12-15-2010.csv - this is the malformed integrity report. Service Configuration: Service Mode: Verify the integrity of a list of items Get integrity information from...: The storage provider Salt: empty Store: AMAZON_S3 (0) Space with input listing: admin Input listing name: usecase4-service-test-1.md5 Output space: admin Output listing name: fingerprints-12-15-2010.csv Output report name: integrity-report-12-15-2010.csv -------------------	1
183	DURACLOUD-319	Duplicate on Demand: Service failing	Duplicate on Demand service failed on 12/7 Service Configuration: Source Space - synctest4 Copy to this store - RACKSPACE (1) Copy to this space - replicants Store results file in this space - replicate-out Working Space - tmp2 Number of Server Instances - 1 Type of Server Instance - Large Instance Service Final Status: Job Ended - 2010-12-07 at 04:13:29 PM UTC Service Status - Job Complete serviceId - replication-on-demand-service-0.7.1 serviceStatus - STARTED Job State - FAILED Job Started - 2010-12-07 at 03:14:37 PM UTC Job ID - j-9PDVC8SADTPQ Additional Statistics: - Run at icpsr.duracloud.org - synctest4 (source) space has: 74993 items (it is a closed space) - tmp2 (work) space has: 84 log files (all generated by one run of the service) - job id j-9PDVC8SADTPQ ran for: 57 minutes before failing - MapReduce job failed after 57 minutes - 2 files were successfully duplicated - In logs: /steps/1/stderr shows 10 task attempts which were all killed after not responding in 10 minutes	2
184	DURACLOUD-320	Duradmin upload failures for multiple large files	When uploading multiple files through duradmin, failures frequently occur causing the content to not land in durastore. scenario: 1. select a ~100-MB file 2. upload it five times in a row through duradmin using different content-ids, and not waiting for one to complete before uploading them all 3. verify all five files successfully upload The scenario above consistently fails.	1
185	DURACLOUD-323	Bulk Bit Integrity Service: add comparison	The bulk bit integrity service should contain the following updates: - add optional config option that takes a user-specified input listing that is used to verify MD5s after the hadoop servers have generated a new listing. -- if no listing is provided, the service should compare against MD5s stored in storageprovider metadata -- this post processing should be implemented as a PostJobWorker	3
186	DURACLOUD-324	reverse upload history order in the upload window	It would be nice to reverse the order of the upload history in the upload window, so that more recent and/or in-progress uploads appear at the top instead of the bottom of the popup window. Sort of what the Firefox download window does. Putting the recent ones at the bottom requires the user to scroll down to see the progress bars of the recent uploads.	1
187	DURACLOUD-325	Image Viewer Flyout no longer works	In the past, a content item that had an 'image/*' mimetype would present a link on the image thumbnail (if djatoka service was running) or on the canned picture icon. This link would display a "flyout" image viewer. Now, clicking on the thumbnail or picture icon has no effect.	1
188	DURACLOUD-326	DurAdmin incorrectly displaying 0 for space size	When using DurAdmin to display spaces which are larger than 1000 items, the space size is often mis-reported as being 0, even though there is a listing of items in the space. Also, clicking on the space no longer refreshes the item listing or metadata of a space (this is also true for content items.)	1
189	DURACLOUD-327	DurAdmin: Delete button mis-aligned	Hovering over a multi-line content item displays a delete button which is half-way between the current item and the next item in the list. The delete button should be displayed completely within the boundary of the content item which it will delete.	1
190	DURACLOUD-329	Video streaming in duradmin:spaces	This item is to address the issue of duradmin not showing the streaming video when clicking on a content item in a space on which the MediaStreamer has been enabled. Scenario: # create two spaces: 'video' & 'tmp' # add the mp4 file attached to this jira item to the space 'video' # deploy the MediaStreamer server using 'video' as source and 'tmp' as destination # wait until service deployment is complete # click on the mp4 item in the 'video' space # ! the video should be shown in the right-hand column at this point ! Note: If you go directly to https://<host>/durastore/tmp/singleplayer.html the streaming video will be available.	1
191	DURACLOUD-331	NullPointerException with Duplicate-on-change service on 'delete content'	This item is to address the issue of a NPE being thrown when the Duplicate-On-Change service is running and an item is deleted. Monitoring the duracloud-servicesadmin.log will show that upon the above action, an NPE is thrown in: org.duracloud.services.replication.Replicator.replicateContent(Replicator.java:127) The service does successfully remove the item from the secondary store, and remains alive in the OSGi container; however, this case should probably be guarded against.	3
192	DURACLOUD-332	Duplicate on Change: space access is not propagated	This item is to address the issue of the Duplicate-on-demand service not updating access state on the secondary provider when a space at the primary provider is changed from open to closed, or vice versa. Scenario: # deploy the duplicate-on-demand service # create a space in the primary store # verify that the space was also created in the secondary store # verify that both spaces have the same access state # toggle the access state of the primary space # boom!, notice the access state of the secondary space did not change	1
193	DURACLOUD-333	Changing space access removes the space metadata	When changing the access level of a space the space metadata is removed. Create a space Add space metadata Change the space access Refresh the space Notice the space metadata is gone	1
194	DURACLOUD-335	Media streamer: files not getting set with proper permissions	On occasion, when starting the media streamer service, one or several of the files to be streamed is not set to have the proper permissions for streaming. Thus far, this issue has always been resolved by restarting the media streamer. It seems likely that the missing permission is due to a failed call to S3, so a retry would likely fix the problem.	1
195	DURACLOUD-338	Chunker Tool: error pushing files in sub-directories on Windows	When running the chunker tool on Windows, if the content directory includes sub-directories with files, the tool will attempt to send each file to DuraCloud with a name "sub-dir-name\file-name", which causes an error because the "\" character cannot be part of a content ID. This does not show up on Linux/Mac systems because the "/" is used instead. This task is to update the chunker tool to replace "\" characters with "/" characters in all generated content IDs.	1
196	DURACLOUD-339	DurAdmin: Errors on space filter	When attempting to filter by name in a space (used the x-service-work dir as an example, need this features to get logs for a specific bulk run), I tend to get "next page failed: error" and "Get Space failed: error" pop-ups. The filter works and loads the list of filtered content items correctly, so the errors seem unnecessary. It appears that a new attempts is made to get the content listing on every change to the filter box, so each letter I type creates a new call. My guess is that just adding a "Filter" button beside the text box, and only making the call when the button is pressed would fix the issue.	1
197	DURACLOUD-341	Duplication on Demand destination space	When configuring the Duplication on Demand service, if you leave the destination space name blank, the service should automatically create the destination space with the source space's name. Carissa ran the service leaving the destination space blank (assuming this would occur) and the service ran to completion... service status read: "complete" but no duplicated content was to be found.	1
198	DURACLOUD-342	Bulk Image Transformer output to be standard CSV format	Currently the output for the Bulk Image Transformer is in a modified CSV format, similar to: name1=value1, name2=value2, name3=value3, ... This task is to update the output to be in standard CSV format: value1, value2, value3 The first line in the file should a header which indicates the names of the values to follow: name1, name2, name3 The Bulk Bit Integrity service produces this format by allowing the hadoop reducer to produce all of the value lines, then using a post processor to insert the name line at the top.	2
199	DURACLOUD-344	Consistency of service output destinations and names	This improvement is to ensure that all report-like outputs from services are created in the x-service-out space. Also, the outputs should have a consistent naming convention: <service-identifier>/<report-name> where <service-identfier> = all lowercase with '-' between words if necessary	4
200	DURACLOUD-346	Refactor integration tests that regularly fail	This task is to determine the integration tests that regularly fail due to "eventual consistency" effects or other issues related to the underlying storage provider implementations. Eventually these tests should either be refactored to be more resistant to underlying storage issues, or should be refactored into unit tests that do not require network connectivity. However, this refactoring work will be addressed in a separate task. https://jira.duraspace.org/browse/DURACLOUD-237	5
201	DURACLOUD-347	Hashed user credentials	This improvement is to change from the current pattern of storing user credentials in-memory as plain text to, storing the hash of the credentials in-memory and having authN perform the same hashing (SHA256?). See Spring Security for adding a "password-encoder" to authentication providers in both 2.x and 3.x versions.	5
202	DURACLOUD-348	Bulk service status message	This improvement is to more accurately display the service status state of bulk services. Currently, bulk image conversion and bulk bit integrity checking are processed in two steps: 1. hadoop processing 2. post processing on duracloud instance The service status shows conflicting messages. The "Job State" is "COMPLETE" when the hadoop step is done, but the "service status" shows "Post Job Processing" for bit integrity, and nothing for image conversion. More accurate and consistent status messages need to be put in place across the services.	2
203	DURACLOUD-349	Bit Integrity Service: iterating over http stream	This issue is to change the ListingIterator found in HashFinderWorkload.java to iterate over a cached file inputstream rather than the content inputstream from an http GET. Although in most scenarios this is not an issue, when the content stream is large (30,000+ lines) the http stream sometimes breaks, causing the iteration to stop.	1
204	DURACLOUD-351	Allow build-time specification of dependency osgi bundle version	This improvement is to allow services built within a particular version of the baseline (say 1.4.0) to specify compatibility with dependency DuraCloud-osgi bundles from another version (say 1.3.0).	1
205	DURACLOUD-352	Bit Integrity Service: Support comparison for space manifests of up to 1 million items	This improvement is to ensure that spaces that contain up to 1 million items can successfully perform a bit-integrity manifest verification.	2
206	DURACLOUD-353	Bulk Bit Integrity Service: shutdown of post-processor	This item is to address the issue of shutting down not only the Bulk Bit Integrity Service on "undeploy", but also shutting down the post-processor. To reproduce this error, start the BBIS over a space containing more than 1000 items. Once the "Job State" indicates "COMPLETE", but while the "Service Status" still indicates "Post Job Processing...", undeploy the service. Undeploy fails due to the inability to remove the fixityservice OSGi bundle from the services container, as it is still processing. At this point, the BBIS remains in duraservice, but is no longer available via servicesadmin.	2
207	DURACLOUD-354	Bulk Image Conversion: failure due to Hadoop AMI upgrade	This item is to address the breakage caused by amazon elasticmapreduce changing their default AMI for hadoop jobs. This effects the bulk image conversion service which is faulting in the bootstrap script due to a missing dependency in the apt-get initialization.	1
208	DURACLOUD-355	Media Streamer: Allow streaming of multiple spaces	This task is to allow the user to select multiple spaces for the media streaming service to stream.	2
209	DURACLOUD-356	Synctool: selective sync-ing	This improvement should allow the user to selectively sync specific files based on subdirectories, prefixes, and/or suffixes using the synctool.	3
210	DURACLOUD-357	Media Streamer: Stream files added to a space	This task is to set up a listener for new files which are added to spaces on which streaming is already enabled, and enable those files for streaming as well.	4
211	DURACLOUD-358	Revisit build from source	It appears that an oc4j dependency used by pax-construct is no longer available in a public m2 repository. This task should revisit pax to see if an upgrade is in order and to ensure that the build works from scratch.	5
212	DURACLOUD-364	Single Use Access URLs	This task is to allow for the creation of single use access URLs. The user would need to specify the file which they would like to provide access to, then DuraCloud would provide a link by which that file can be accessed a single time. After the URL is used once, it no longer allows access to the file.	5
213	DURACLOUD-365	service reports contain another differentiator	Service reports that appear in the service-output space need to have an additional differentiator so that services that are run multiple times in one day do not overwrite one another.	4
214	DURACLOUD-369	Bulk Bit Integrity Service: post-processing in hadoop	This improvement is to change the bulk bit integrity service to run its MD5 metadata collection post-processor in hadoop instead of with a single process on the main duracloud instance. This enables the service to run within hours instead of days for spaces exceeding 50,000 items.	3
215	DURACLOUD-370	Notification/Email utility	This feature is to expose a utility for sending emails. It should consist of both a general interface project and a corresponding implementation project.	2
216	DURACLOUD-371	Remove interim bulk-bit-integrity manifests	This improvement is to remove from the service output space the interim manifests that the bulk-bit-integrity service creates once the final report has been created. These manifests are: bit-integrity-metadata-results.csv bit-integrity-results.csv	3
217	DURACLOUD-374	Service Report: backend utility for currently running services	Provide a way to generate a report containing: - details from running services The details should include at least the following information: - tbd The reports should also be available for retrieval through a REST-api	5
218	DURACLOUD-375	Storage Report Dashboard	This task is to create a UI dashboard for displaying and downloading reports on storage. REST API calls should be available from which the storage report information can be gathered. A UI will need to be created to be able to display the information in the reports. The reports will likely be in XML, so XSLT is an option for display.	5
219	DURACLOUD-376	Limit time that can be spent spinning on a mapper task within hadoop	Currently, the service code which runs in hadoop kicks off a thread to ensure that the mapper has enough time to complete. These threads are not time limited, so if the mapper process hangs, the hadoop job could potentially live forever. This task is to revisit these threads for each of the bulk jobs to set a limit for the amount of time the threads give to a mapper activity.	2
220	DURACLOUD-377	Remove user management from Duradmin	This task is to remove the user-management capabilities ("Adminstration" tab) from Duradmin, since this functionality is being moved up the the account-management-app.	1
221	DURACLOUD-379	Run-away service monitor	This task is to create a script/application that monitors DuraCloud accounts for long-running bulk services and sends an email alert upon detection.	1
222	DURACLOUD-380	Azure Storage Provider - adding content and specifying a checksum	When a checksum is provided upon adding content to Azure, ChecksumInputStream was incorrectly trying to calculate the checksum from the provided stream. The provided checksum should be used instead of calculating it from the stream.	1
223	DURACLOUD-381	DuraCloud instance losing initialization information	1. Restart a Duracloud instance via the Amazon Management Console 2. Wait 5 minutes 3. Run app-config to initialize the instance 4. Check the instance, everything looks good 5. Wait between 7-10 min, check again - this time none of the users that were in the init are in Duradmin or Duraservice After all of these steps, if you log in to DurAdmin as root, you will see on the Admin tab that no users are listed, and going to the Services tab brings up a "get available services failed: error" dialog. You will also not be able to log in to DurAdmin as any user besides root. Interestingly, everything seems normal on the Spaces tab, and rest calls to durastore work fine. As a secondary issue, if on step 2 you don't wait 5 minutes, but go ahead when each of duradmin/durastore/duraservice are available (less than 2 min after restart), then in a short amount of time after you init, (less than 2 min) the durastore and duraservice init go away. But in this case all of the users are still present.	5
224	DURACLOUD-382	Create Example ServiceClient	This task is to create a sample ServiceClient analogous to the Sample StoreClient.	1
225	DURACLOUD-383	Bit Integrity Service: support retries of interrupted content downloads	This task is to add retries to the bit-integrity-service when a content item that is downloaded to calculate its MD5 does not match the expected MD5 from metadata.	1
226	DURACLOUD-384	Bit Integrity Service: Pass into comparison phase content-id of csv input	The comparison phase of an 'all-in-one-for-space' run of the bit-integrity-service tries to guess the content-id of the input csv files. With date/timestamps appended to the content-ids, this becomes difficult. Therefore, the input content-ids should be passed into the comparison phase.	1
227	DURACLOUD-385	Image Viewing URL Stability	This task is to investigate the stability of image viewing URLs across upgrades of DuraCloud instances. As each DuraCloud instance has an allocated static I.P. address, the behavior is unexpected. If there are specific uses cases for demonstrating this issue, please add them as comments below.	2
228	DURACLOUD-387	Error on large space delete	This item is to resolve issues seen when trying to delete a large space On the RackSpace storage provider: In the bhl.duracloud.org instance, there is a space "master-rack" that initially contained ~26,000 items. Deleting single items or a page of 200 items succeeded; however, when trying to delete the entire space a pop-up pops after a few minutes indicating that the operation failed. After one attempt, the number of items dropped to: 25,295 After another attempt, before failing the number of items dropped to: 24,727 In looking at the logs on the instance, it appears that two different types of errors were occurring, both of which were coming from RackSpace: - Could not connect to Rackspace due to error: Read timed out - Object was not found reportofscientif03scotrich/reportofscientif03scotrich_metasource.xml (other content-items also reported as unfound) See attached catalina.out for details. On the S3 storage provider: In the bill.duracloud.org instance, there was a space "num-files-test-work-3" that initially contained 36,845 items Clicking on "Delete Space" in DurAdmin started it spinning for about 5 minutes, it then came back with a dialog saying "Failed to Delete Space". Checking on the space size after the dialog popped up showed it at ~31,000 items However, even with DurAdmin out of the loop, the size of the space continued to drop and the space was eventually deleted, after about 35 minutes There were no errors in the logs indicating a failure.	5
229	DURACLOUD-389	Service Registry Naming	This update is needed to support creating a service-config.xml document that is named the same as the allowable service registry name. The old convention was: duracloud-0.9.0-service-repo The new convention is: duracloud-0-9-0-service-repo	1
230	DURACLOUD-390	Security over UserDetailsService.getUsers()	This improvement is to add method security over UserDetailsService.getUsers() to limit access to this call to ROLE_ADMIN.	1
231	DURACLOUD-392	Create installation package for release	Such an installation package should include: - documentation - the three wars (duradmin, duraservice, durastore) - zip of service registry - zip of osgi container	1
232	DURACLOUD-394	Page load latency	The Duradmin pages are built using jQuery. When the browser loads, the scripts pull data while building the page, creating a perceptible delay in rendering. A recommended solution may be to pull some of the jQuery back and not try to build the screen using as much scripting. Browser versions and usage scenarios need to be added to this jira item.	2
233	DURACLOUD-397	Metadata editor allows for blank values	Metadata editing via DurAdmin (using both the regular editor and the "Edit Metadata" dialog) allows you to add empty name/value pairs to the metadata list. Metadata names should be required. Although the UI appears to allow empty metadata name/values, if the name is empty, the metadata does not persist.	1
234	DURACLOUD-398	UI shortcuts which are not useful should be disabled	Holding shift + one of the arrow keys does odd things to the display. These shortcuts should be disabled.	1
235	DURACLOUD-399	Allow user to select which spaces to include when running Duplicate on Change	This task is to provide the option for users to select a list of spaces when configuring the Duplicate on Change service, then only perform duplications for items in those spaces. One reason for this feature is to allow users to choose to not duplicate x-service-out and x-service-work spaces.	3
236	DURACLOUD-400	DuraStore REST API: text/plain response from getSpaces	Just an inconsistency I noticed while looking at the REST API. This particular method gives a response Content-Type header of "text/plain". Most other methods that return XML declare their Content-Type as text/xml. For example: https://demo.duracloud.org/durastore/spaces?storeID=0 ...gives text/plain, but: https://demo.duracloud.org/durastore/cwilper-test?storeID=0 ...gives text/xml.	1
237	DURACLOUD-401	Use application/xml as response Content-Type for XML-returning REST API methods	Most of the REST API methods that return XML today use text/xml, but application/xml is a actually a better choice due to character encoding issues: According to the specification of the text/* media types (RFC2046), if a charset parameter is not specified in the declaration of mime type (at the end of the Content-Type value, ala "text/xml; charset=utf-8", it must be assumed that the body is encoded in US-ASCII. The "encoding" declared in the XML declaration is supposed to be ignored. Here are a couple online posts that further explain the problem: * http://annevankesteren.nl/2005/03/text-xml * http://www.grauw.nl/blog/entry/489 It's clearly the intent that DuraCloud REST API methods that return XML are encoded in UTF-8 (because the xml declaration says so), but according to the standards, because text/xml is used instead of application/xml, and the Content-Type is not declared with a "charset=utf-8" MIME parameter, the xml declaration is supposed to be ignored by clients and they should assume US-ASCII.	1
238	DURACLOUD-402	Allow for local service repository	It is currently required that the service repository for a DuraCloud instance be hosted on another DuraCloud instance. This task is to allow the service repository to be hosted within the same instance. This will both simplify the deployment of stand-alone DuraCloud instances (for users of the open source platform) and lead the way toward allowing multiple service repositories to be associated with a single instance.	1
239	DURACLOUD-403	Improve binary installation package	In the 0.9 release the first binary installation package was created. This allows system admins to deploy an instance of DuraCloud without needing to build from source. This was a good first step, but there are a few pieces which could make the process smoother: - Include wars with names that do not include version. When the jars are dropped into tomcat with the release version included in the file name, they are not deployed to the expected context path (i.e. rather than a context path of /durastore it is instead /durastore-<version>. The installation package zip file includes the version number, this should be sufficient. - Include the app-config jar, as this is needed to configure the instance. - Include an example properties file to be used with app-config.jar for instance initialization. - Include a default/example logback.xml file which can be used within the service bundle home. - Update included run.bat file to properly pick up environment variables in a Windows environment - Allow for the use of a local service repository (handled by DURACLOUD-402) - Update the included readme.txt file to have complete instructions for standing up a DuraCloud instance using the files provided with the installation package. Parts of this document can refer to the wiki for further explanation, but ideally, the entire process would be well enough defined in the readme such that the links to the wiki would be optional reading.	1
240	DURACLOUD-404	Duraservice: clean re-initialization	Currently, when re-initializing duraservice, due to caching, some elements do not take on the new re-initialize values. This improvement is to determine and implement what is required to make duraservice re-initialization work as if it were being initialized from scratch.	1
241	DURACLOUD-406	Service.xml: generate on build, not on test	This improvement is to have the services.xml service-repo manifest created during the mvn 'package' phase instead of during 'test'.	1
242	DURACLOUD-407	Service.xml: generate various renditions	This feature is to have the services.xml service-repo manifest generated for each of the various DuraCloud account package types: - archiving / preservation - media access - professional - trial	1
243	DURACLOUD-408	Host DuraCloud maven artifacts with Sonatype	This item includes the following tasks: - open ticket with sonatype for org.duracloud - check if deps are outside of central - check dep licenses - make sure our poms are of good quality (sonatype oss) The following plugin will be useful. mvn dependency:tree https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide	5
244	DURACLOUD-409	Failed to copy all files when running a large Duplicate-on-Demand service	A small percentage of files are not copied during the execution of a large Duplicate on Demand service execution. This can result in either a missing file or a zero length file in the target space. This has bug has been demonstrated repeatedly both between different providers and copies back to the same provider. However, it has only been observed in large file sets (circa 70,000).	16
245	DURACLOUD-410	Use Amazon multi-part upload for large files	This task is to update the S3StorageProvider to make use of Amazon's multi-part upload option for S3 when content is large enough to require chunking. This would mean that rather than storing our chunked file parts (and associated manifest) in S3, we would use the multi-part tools to push up the pieces and assemble them again in S3. Since the MD5 for the entire chunked file is not available from S3, we will need to calculate it on transfer and store it as metadata.	5
246	DURACLOUD-416	Implement a content cache to mitigate eventual consistency issues	In order to make the issues of eventual consistency an issue that does not concern the users of DuraCloud, DuraStore should be updated to provide a cache for the content which is stored. This could mean either storing the file locally first, then making the call to push it to the provider, or wrapping the stream in order to write the file locally while it is being stored at the provider. These local files would be used to fulfill GET requests that come in shortly after the file has been stored. The files in the cache would be removed once there is a greater degree of certainty that the file will be available on all following requests to the provider (perhaps just after a certain time period has elapsed.) Note that one benefit to storing the files locally prior to pushing them to the provider would be that we could know the file size and generate the checksum locally before making the provider request, meaning that we could ensure that the file reached DuraStore correctly before passing it along to the next location. This may also become useful for handling large files that we intend to pass to the provider using a multi-part upload.	5
247	DURACLOUD-420	Create 1.0.0 AMI	Updates to the 1.0.0 AMI should include: - trim down apache rules to only expect j2k to be running on port 18080 - add UTF-8 support to tomcat connectors <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8" /> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" /> - add debug tomcat startup script	1
248	DURACLOUD-423	Filter display of service properties	This feature is to omit the rendering of service properties that have names prefixed with "System ". These are service properties that are useful for monitoring and scripting, but are probably confusing for end-users of Duradmin.	1
249	DURACLOUD-424	Service completion state must be accurate: non-bulk services	This improvement is to reflect the completion state of non-bulk services as "success" or "failure". Currently, the service state remains "started". Affected services: bit integrity checker bit integrity checker tools image transformer Additionally, the end-time of the service should be captured in the service properties.	3
250	DURACLOUD-425	Service completion state must be accurate: bulk services	This improvement is to reflect the completion state of bulk services as "success" or "failure". Currently, the service state remains "started". Affected services: bit integrity checker bulk image transformer bulk duplicate on demand Additionally, the end-time of the service should be captured in the service properties.	2
251	DURACLOUD-427	WayBack Machine Service	For DuraCloud users that are storing web archive content (warcs), a WayBack machine for search and access would be valuable. This feature is to create such a service. reference: https://webarchive.jira.com/wiki/display/wayback/Wayback+Installation+and+Configuration+Guide	8
252	DURACLOUD-429	iRODS/Chronopolis Storage Adapter	This feature is to add a new storage adapter to connect to iRODS/Chronopolis, ala S3, Rackspace, & Azure. The base code has already been provided by Mike Smorul.	3
253	DURACLOUD-430	Bulk upload client tool with UI	This task is to create a client-side tool which provides a nice user interface for managing bulk upload tasks. This tool should provide a UI that allows files to be uploaded via the following methods: - file/directory selection dialog A progress bar should be provided which indicates the status of the upload, specifying which file is being uploaded and the progress of the transfer. It may make sense to use the SyncTool in the background to manage file transfers.	8
254	DURACLOUD-431	Duradmin mimetype editor: 'application/atom+xml' regex	When attempting to set the mimetype of a content item to 'application/atom+xml', the error "Invalid Mimetype" is displayed. This prohibits the setting of the desired mimetype.	1
255	DURACLOUD-432	file name collisions not detected in Google Chrome	When using DurAdmin with Google Chrome to upload a file, there is no file name collision detection or pop-up warning. The upload simply replaces the file. I am using Google Chrome on my mac os x. Note that the file name detection and pop-up did work on Firefox.	1
256	DURACLOUD-433	Duradmin service status	When a service is deployed via duradmin, an entry is created detailing the: - service - hostname - status The 'status' is apparently populated when the service is deployed and never updated as the actual service status changes through the course of processing. This improvement is to keep the status info in the UI entry up-to-date by pulling the content from the "Service Status" property instead of from the Deployment.status The current/accurate status info is also displayed on the right hand side, under the service details.	1
257	DURACLOUD-435	Reinit should clear the cached StorageProviders	Upon reinit the cached StorageProviders should be cleared in order to specify different stores	1
258	DURACLOUD-436	Service registry name and service.xml dependency	This improvement is to decouple the dependency on the content-id of the service.xml file of a service registry from the space-id of the registry itself. Currently, the content-id is required to be identical to the space-id with the addition of the '.xml' extension. The service.xml content-id should be allowed to be any name.	2
259	DURACLOUD-438	Add service-result to completed service properties	This improvement is to add a new service property across all services that have a completion (success or failure) that notes the content-id of the service result file. This property should be set upon service completion.	1
260	DURACLOUD-439	Enforce space id rules for services	Enforce space id rules when entering a space name for Duplicate on Demand and any other service	1
261	DURACLOUD-440	Handle all unicode characters for storing and retrieving content	This task is to ensure that a content item can be added and retrieved from both DuraStore and DurAdmin using any characters that can be expressed via unicode. Generally, it seems like it's possible to add content items with any unicode content ID, but the problems come in when attempting to access/update/delete those items. These problems are not consistent between DuraStore and DurAdmin. Occationally DuraStore allows for retrieving files that DurAdmin does not. A few Ascii content IDs which currently fail this test are below. This list is not exhaustive, just a examples. test#item - duradmin cannot load content - error when accessing via durastore test%item - duradmin cannot load content - bad request response from durastore test&item - duradmin cannot load content - works fine in durastore test+item - duradmin cannot even bring up metadata - works fine in durastore Of course, content IDs with non-ascii characters also have problems: 1. blankHélènåJör.txt 2. blankπ × { } © 佈 б ' ' —.txt #1 seems to be only a problem in DurAdmin, the file can be retrieved and deleted via DuraStore with no problem. #2 is a problem for both DurAdmin and DuraStore, the file cannot be retrieved or deleted via either.	5
262	DURACLOUD-441	ServiceManager and ServiceManager Client to share Interface	This improvement is for the following two classes to implement the same interface: org.duracloud.client.ServicesManager.java org.duracloud.duraservice.mgmt.ServiceManager.java	1
263	DURACLOUD-442	Change "metadata" to "properties"	This task is to transition away from the use of the term "metadata" in regard to name-value pairs that can be associated with DuraCloud spaces and content. Instead, the term "properties" will be used. The work involved will be in transitioning between the two terms throughout the baseline.	2
264	DURACLOUD-443	Service Report: create schema and serializer for ServiceReport.getCompletedServicesReportList()	This task is to create a schema and serializer for the list of content-ids returned from ServiceReport.getCompletedServicesReportList(). The expected xml should look something like this: <service-report-ids> <service-report-id>Service-Report-2011-06-27.xml</service-report-id> <service-report-ids>	1
265	DURACLOUD-444	Cumulative storage report	This task is to add an additional storage report that is a cumulative on-going report which is updated each time a new storage report completes. The information in this report will be an entry for each completed storage report that provides the top level report information. Mime type information would be left out, and other information in the detailed report may be excluded as well. The goal of this report is to be able to create graphs that show the change in content stored over time. This report could work like the completed services report in that it could be cycled to start a new file monthly. Information should be included in the cumulative report that allows for the creation of links to the detailed reports as well as cross-links to the detailed report charts. This could just be the detailed report content ID.	3
266	DURACLOUD-445	StorageProvider support for object copy/rename	This improvement is to provide the StorageProvider interface support to: 1. copy an object stored within DuraStore to another space within the same storage provider 2. rename an object stored within DuraStore to a new content-id within its existing space All three storage providers (Amazon, Rackspace, Azure) provider REST-API support for 'copy', but it appears that only the Amazon and Azure java clients expose this functionality. http://docs.rackspacecloud.com/files/api/v1/cf-devguide-20110629.pdf http://docs.amazonwebservices.com/AmazonS3/latest/API/ http://msdn.microsoft.com/en-us/library/dd894037.aspx	4
267	DURACLOUD-446	Service Reports: Comma-delimited content-ids	This bug is that service reports become invalid if a content-id contains a ',' since currently service reports are output as comma-delimited. A potential solution is to create tab-delimited service reports.	2
268	DURACLOUD-447	Update DuraReport storage reporting XML to be generated using JAXB	The purpose of this task is to swap out the current XStream implementation of the DuraReport storage report generation to use JAXB. The main reason for this is to be able to provide an XML schema for the data that is being generated, so that the information can be more easily retrieved and used.	3
269	DURACLOUD-450	Link Service Report in service properties display	This improvement is to add a link to the services "Report" property to display (possibly as a flyout?) a tabular form of the service results.	2
270	DURACLOUD-451	UI: Use different icons for each service	The goal of this task is to include a different icon beside each service, both in the deployed services listing and the available services listing.	2
271	DURACLOUD-452	UI: Remove the provider drop-down when there is only one provider	This task is to remove the storage provider drop-down when only one provider is enabled.	1
272	DURACLOUD-453	UI: Use different icons for the content item Download and View links	This task is to replace the button icon for the content item View link, as it is currently the same as the download link. Perhaps a eye icon or something similar would work.	1
273	DURACLOUD-454	UI: Adjust the service deployment box to remove scroll bars	This task is to re-adjust the pop-up dialog used for deploying services so that scrolling is not necessary and no scroll bars are displayed.	1
274	DURACLOUD-455	UI: Display service status correctly	Currently, the table which lists all deployed services has a column called Status which always displays STARTED, regardless of the actual status of the services. This value is being read from "deployment status" which is meaningless. This task is to read in the actual status of each service from the service properties and display that in the table.	1
275	DURACLOUD-456	UI: Administration tab always contains text: "No users defined"	This task is to only display the "No users defined" text on the Administration tab when there are actually no users defined.	1
276	DURACLOUD-457	Service Monitoring on reconfigure	This improvement is to ensure that a monitor thread is in place when a service is reconfigured. A service can be reconfigured while it is either still running, or after it has completed. Since a reconfigured service retains the same deployment-id, a service that is still running will continue to be monitored by the original monitoring thread. However, a service that is complete needs to have a new monitor thread started since the service is effectively restarted via a reconfiguration.	1
277	DURACLOUD-458	Space Delete: Indicate to users when a space is in the process of being deleted	Currently, when deleting a space of reasonable size there is a dialog that spins for a while, then the space is removed from the spaces list. But if the list of spaces is refreshed, the "deleted" space shows up again. In the background a process is still running to delete the space, but there is no way for the user to know that this is happening. This task is to make it clear to users when a space is in the process of being deleted.	2
278	DURACLOUD-460	Provide informative error messages for special characters in space and content properties	Special characters included in metadata names or values which are added to spaces or content show up immediately in the list of metadata/tags, but when those special characters aren't valid http header values, the information is not actually added to the space or content item in the provider. This task is to to check for valid characters and pass back errors to users so they know the allowable characters.	3
279	DURACLOUD-462	UI: Inform users that the characters '\', '?', and ';' cannot be part of a Content ID	Three characters are not allowed in content Ids: '\', '?', and ';', as noted in IdUtil and here: https://wiki.duraspace.org/display/DSPINT/Space+and+Content+Name+Restrictions. Errors are returned from DuraStore when trying to add an item with a content Id including '\' or '?' characters, but no useful information is presented to users in DurAdmin. No error is returned with the use of the ';' character, as the character doesn't get passed through to DuraStore. There should be a validator in DurAdmin to catch these characters, and users should be provided a useful error message indicating the problem.	1
280	DURACLOUD-464	UI: XML files viewed in Chrome are appended with javascript code	When using DurAdmin via the Chrome browser, selecting a file of type XML, and choosing to View the file, the XML is rendered correctly, but is followed by a block of javascript code, starting with "try{function lpshowmenudiv(id)".	2
281	DURACLOUD-465	Handle semicolon character	When content whose ID contains a semicolon (;) character is added on a DuraCloud instance, the characters after the semicolon are dropped. The file is available, but the ID is wrong.	2
282	DURACLOUD-466	DuraCloud Constants	This task is to create a new maven project that simply holds constant values required across the other projects. Likely, these constants should be implemented as enums. Examples include: - DELIM = '\t' - ENCODING = "UTF-8" Other candidates include service constants (see: org.duracloud.services.ComputeService.java) and other common, constant, classes that have no other dependencies such as: org.duracloud.common.util.DateUtil.DateFormat.java	2
283	DURACLOUD-467	UI: Using Enter key on Add Space selects Closed space access	Using the Enter key on the Add Space dialog to submit the request to add a space both selects the Closed access button and submits the request. The space is created correctly, but the access button indicates the default Open state before the Enter key is hit, then the space ends up with Closed access after it is created. Tested on: Windows 7; Firefox 5.0; Chrome 12.0.742.122	1
284	DURACLOUD-468	UI: Add message to indicate when no other services are available to deploy	When all services are deployed, and you click on Available Services, there is no message that there are no services available	1
285	DURACLOUD-469	UI: Display the selected service on the service configuration dialog	The name of the service which was selected needs to be displayed on the screen used to configure the service. Currently there is just text that reads "Configure the Service", which isn't very helpful.	1
286	DURACLOUD-471	Document Installation Process	This task is to update the trunk/resources/readme.txt file to have complete instructions for standing up a DuraCloud instance using the files provided with the installation package. Parts of this document can refer to the wiki for further explanation, but ideally, the entire process would be well enough defined in the readme such that the links to the wiki would be optional reading. The initial outline of the process should be documented by one of the developers, then the detailed steps should be flushed out by someone who is not as familiar with the process.	2
287	DURACLOUD-474	Service Errors with removal of expected spaces	This task is to analyse the exact impact of removing the spaces: x-service-out x-service-work before and during a run of each service. After the analysis, implement a strategy for allowing services to run successfully to completion. Resolutions could potentially include: - services exiting gracefully - services recreating the expected spaces - or potentially not allowing the spaces to be deleted in the first place	3
288	DURACLOUD-475	Create consistent css styling for reporting	This improvement is to restyle the Dashboard Storage and Services tabs to have a consistent styling with the rest of the duradmin application.	3
289	DURACLOUD-478	Facilitate integration: CDL Merritt	This task is to support the conversations and integration of Merritt with DuraCloud. Likely the software development will all be on the CDL side, but some time and effort will be needed as support.	1
290	DURACLOUD-481	CloudSync Service	This feature is to add a new CloudSync service. The service will likely be a webapp service (ala j2kservice). Note: CloudSync itself should probably by updated to take its working directory as an initialization argument.	2
291	DURACLOUD-482	Re-Add DuraCloud favicon	This update is to re-add the DuraCloud favicon to the duradmin application. In release 1.0.0, its rendering was somehow dropped.	1
292	DURACLOUD-484	DurAdmin support for object copy/rename	This feature is to add UI support for single content copy/rename operations.	3
293	DURACLOUD-485	StoreClient/REST support for object copy/rename	This feature is to expose the StorageProvider.copyContent() method through the DuraStore REST API as well as update the storeclient. However, this feature should not be implemented until the patch submitted to Rackspace has been incorporated into the Rackspace java-client baseline. https://github.com/rackspace/java-cloudfiles/pull/18	3
294	DURACLOUD-486	Duplicate on Change may not successfully duplicate all files	In testing completed by Harvard, the Duplicate on Change service was deployed, followed by running the sync tool to transfer 33,110 files into a single space. The Duplicate on Change service was configured to watch the primary Amazon provider and copy all files to Rackspace. After all files were uploaded and the Duplicate on Change service was given ample time to complete, only 33,045 files had been moved to the space in Rackspace. So 65 files (0.2%) were not duplicated. This task is to determine how this small set of files may have slipped through the cracks, and update the code such that: 1. No files are missed in future runs of the service. 2. An output report is generated which indicates all files that have been transferred and captures any errors that occur along the way. The log files from the harvard instance are attached to this issue for reference.	4
295	DURACLOUD-487	Retrieval Tool: Add the ability to retrieve a space item list	This task is to add an option to the Retrieval Tool which will pull down the list of items in a given space (or list of spaces), rather than actually retrieving them. This list will be written to a file. There is an existing tool in duracloud-utils named the space-lister-tool which already provides this capability, so it may be useful to borrow code from that implementation. This should result in a single new command line option (--list-only perhaps) that, when included causes the space list to be stored in the content directory rather than files.	1
296	DURACLOUD-489	Serve DurAdmin pages as XHTML	This task is to update DurAdmin to serve its pages as strict XHTML. The mime type used should be: application/xhtml+xml. The DOCTYPE header should also be specified as <!DOCTYPE html> which is the recommendation for HTML5.	2
297	DURACLOUD-490	1.1 Production AMI	This task is to create a new production AMI with the 1.1 tagged codebase. It should include: - /etc/init.d/tomcat: export CATALINA_OPTS="-Xms512M -Xmx512M -XX:MaxPermSize=512M"	1
298	DURACLOUD-491	Add version number to output of sync and retrieval tools	This task is to include a version number in the printed output of the sync and retrieval tools so that it's easy to determine which version is being used.	1
299	DURACLOUD-492	StoreClient moveContent()	This feature is to add an additional method to the storeclient, namely, moveContent(). This method will simply call copyContent() followed by deleteContent().	1
300	DURACLOUD-493	Duradmin login url	This issue shows up in the following scenario: - Launch an instance as root user from the Management Console - the login fails (as expected); but typing in the correct root username/password continues to fail - until manually removing the trailing portion of the URL: "login?error=true" It would be ideal if an incorrect login using the pattern as in the MC did not produce this URL in the browser.	1
301	DURACLOUD-495	Move Duradmin to Dojo	https://wiki.duraspace.org/display/DURACLOUDDEV/Notes+on+the+Way+Forward+for+Duradmin	5
302	DURACLOUD-496	ContentStore Duplicate Across Providers	This feature is to provide users a way to copy a single item from one storage provider to another storage provider (or within the same storage provider). The implementation will likely be similar to the ContentStore.copyContent(), but will contain an optional source storeId. This effort should also include exposing the cross-provider copy functionality at the ContentRest level of Durastore.	2
303	DURACLOUD-497	Remove Image-Conversion Services (temporarily)	This task is to remove the two image conversion services: - bulk image conversion - non-bulk image conversion from the list of available services for the 1.1 release. These services will become available once again when a strategy has been developed to help users manage the error cases.	1
304	DURACLOUD-498	SyncTool does not recognize chunked content	The issue here is seen in two scenarios: 1. Content that is chunked into DuraCloud during one run of the synctool is not recognized as existing in the space on a subsequent run of the synctool. 2. Content that is chunked into DuraCloud during one run of the synctool is not recognized as existing locally on a subsequent run of the synctool, and if the --sync-deletes flag is set, then the chunks are not deleted from the DuraCloud space.	2
305	DURACLOUD-499	Update EasyMock library to version 3.0+	EasyMock has rolled the class extension project under the base project and deprecated EasyMock ClassExtension. This task is to update the top level pom to use EasyMock 3.0+ and refactor all tests which use org.easymock.classextension.EasyMock to use org.easymock.EasyMock instead.	1
306	DURACLOUD-500	SyncTool does not delete empty subdirectories	The issue here can be seen in the following scenario: # create an empty DuraStore space # create a local directory with subdirectories, add files # run synctool ** verify space/directories are the same # delete a local file at the top-level or within a subdirectory (but if in a subdirectory, make sure more content remains in that subdirectory) # run synctool with '--sync-deletes' ** verify content is deleted from the space # here is the issue: delete local content in any of the following ways: ** from a subdirectory, leaving that subdirectory empty ** remove everything from the local directory # run synctool with '--sync-deletes' # Error: notice content is not deleted in space	1
307	DURACLOUD-501	Throw exception on space names which are too long	This item is to ensure that an error is thrown back to the user when a space to be created has a name which exceeds the maximum name limit. Currently the name is just truncated, and the truncated name isn't displayed until a refresh of the UI. See S3ProviderUtil line 133 for the spot where the S3StorageProvider likely does the truncation. All current active storage providers (S3, Rackspace, Azure) should be checked for this same behavior. Example of a space name that is too long (for S3): duracloud-1-1-0-snapshot-service-repo-trial-test	1
308	DURACLOUD-502	put psd files from charles on the wiki	NULL	1
309	DURACLOUD-503	New service plan: All Services	This task is to add a new service plan: "all", that includes all services.	1
310	DURACLOUD-504	OOM on Re-Initialization of Durareport	This issue is revealed when performing multiple re-initialization of a DuraCloud instance via the app-config or MC. It would appear that the report collection threads created during initialization are not properly shutdown on an immediate, subsequent re-initialization. After a few re-inits an OOM is seen.	1
311	DURACLOUD-505	Deployed services list is not refreshing	Once a service has been deployed, the service information is not being displayed in the deployed services list.	1
312	DURACLOUD-506	Add links on the dashboard which allow for downloading DuraCloud utilities	This task is to add a link or links to the DurAdmin dashboard which allow a user to easily download the DuraCloud utilities (sync tool, retrieval tool, etc). It would also be useful to have links pointing to the documentation for each of the utilities that can be downloaded. Update the existing "Help" link to open a new window with this page: https://wiki.duraspace.org/display/DURACLOUD/DuraCloud+Help+Center Add a tab on the Dashboard beside the Storage and Service report tabs called "Tools" with the content listed here: https://wiki.duraspace.org/display/DSPINT/DurAdmin+Tools+tab	1
313	DURACLOUD-508	REST endpoint: isInitialized() across all webapps	This improvement is to add a new REST method to each of the four existing webapps: - durastore - duraservice - durareport - duradmin That authenticates against anonymous users and returns a response code of: - 200 = is initialized - 503 = is not initialized	2
314	DURACLOUD-509	Add messaging support for content copy action	This task is to add messaging support to the new copyContent() method added to the StorageProvider interface. A new topic should be added to carry messages related to copy operations. Once the messages are being sent, the two services which are listeners, Duplicate on Change and Media Streamer, need to be updated to listen for copy operations and react appropriately.	1
315	DURACLOUD-510	Long-running storage tasks	This task is to make it possible to run, track, modify, and stop long-running storage activities. Since these actions can take a long time, the user needs to be able to discover the current progress and also be able to cancel or change the task while it is in process. Also, these tasks usually need to be re-started in the event of an instance restart. Examples of these types of activities: - Multi-file delete - Multi-file copy (within and between providers) - Multi-file or multi-space property add/update/delete - Large space delete For activities which handle multiple content items, there should be a way to provide a known set of content IDs over which the action will occur. A follow-on task for this item will be to expose these capabilities through the UI.	8
316	DURACLOUD-511	UI: Provide an understanable error message when users attempt to create a space with a reserved name	This task is to add validation to the space creation action in the UI which does not allow users to attempt to create a space with a reserved name. The list of reserved names can be found here: https://wiki.duraspace.org/display/DURACLOUDDEV/DuraCloud+Administration. Currently, when a user attempts to create a space named, for example, "task", a big error dialog with a 405 http response message is displayed.	1
317	DURACLOUD-512	Distinguish service-failures from failed-service-results for Hadoop Services	This improvement is to have different informational messages (and underlying service properties) for when a service fails versus when the results of a service run contain failures. Examples of service failures are unrecoverable exceptions, OOM, failed hadoop startup, etc. Examples of result failures are bit integrity discrepancies, unsuccessful image conversions, unsuccessful duplications, etc.	3
318	DURACLOUD-513	Distill failed items from a failed service run	This improvement is to extract out the failed items from the output report of unsuccessful service runs in order to make the errors more clear for the user. The implementation will likely require updates in the reporting UI as well as in the underlying service infrastructure.	3
319	DURACLOUD-514	Filter administrative content from storage reports	This improvement is to by default only expose the user content in the UI storage reports. The administrative content to be filter is currently found in the following spaces: - x-duracloud-admin - x-service-out - x-service-work The implementation may be done at the time of report statistics collection or in the UI, toggling admin spaces on/off.	3
320	DURACLOUD-515	Make parameters consistent among command line utilities	This task is to make command line options as uniform as possible among the current set of command line utilities: sync tool, retrieval tool, chunker, stitcher. There are currently several inconsistencies that need to be cleaned up: - Use -c for content directory (as used by sync and retrieval tools). Currently stitcher uses -d and chunker uses a compound -a option - The chunker should support -i (store-id) - The chunker should use -s (space-id) rather than including it as part of -a - The chunker should use -h (host) and -r (port) rather than -c	2
321	DURACLOUD-516	Service output files to use .tsv extension and text/tab-separated-values mime type	This task is to update the service output files to have mime type of text/tab-separated-values and .tsv extension	2
322	DURACLOUD-517	Undeploy service should have warning	This issue should address the apparent lack of an "ok"/"cancel" warning when undeploying a service. Currently, when the user selects "Undeploy", the service is simply undeployed without a warning notice.	1
323	DURACLOUD-518	Service properties blank on deploy	This issue is to address the situation where a user deploys a service, and when it has successfully deployed, the "Service Detail" panel goes blank. The "Service Detail" panel should show the properties of the newly deployed service.	1
324	DURACLOUD-519	Sync Tool should ensure configuration for subsequent runs is functionally equivalent before attempting to restart	When the Sync Tool starts up, it compares the current configuration with the previous configuration to determine if it can perform a restart. A restart allows the Sync Tool to not attempt to re-sync all of the files in the content directory. The problem is that the check to see if a restart is possible (in SyncTool.restartPossible()) only checks the list of content directories. It should actually compare host, spaceId, storeId, and syncDeletes values as well. If any of those values do not match, a clean start should occur.	1
325	DURACLOUD-520	StorageProvider to support add-content-with-metadata	This improvement is to investigate and potentially implement an update to the StorageProvider.addContent() interface method or an addition of a parallel addContent() method that optionally takes object-properties along with the content-stream. As an example, the current ContentRest.addContent() method performs two calls to the StorageProvider: addContent() & setContentProperties(). The number of calls made to the underlying storage-provider could be cut in half if these two calls were combined. The AmazonS3Client interface, the Rackspace FilesClient, and the Azure IBlobContainer all support calls for including metadata along with the call for adding content.	3
326	DURACLOUD-521	Media Streamer: set logging level down from WARN	This improvement is to set the log level of messages in the MediaStreamService to INFO or DEBUG from the current WARN.	1
327	DURACLOUD-522	StoreClient encryption support	This feature is to provide a means for users to encrypt/decrypt content stored and retrieved through the StoreClient. The StoreClient should have a constructor that optionally takes a symmetric or asymmetric encryption key.	3
328	DURACLOUD-523	SDSC Storage Provider integration	This feature is to integrate the SDSC Cloud as another underlying StorageProvider.	3
329	DURACLOUD-525	Auto-user application	This feature is to provide the ability to start a DuraCloud instance with a set list of services that will be run on a set schedule. The first example of such an auto-run/scheduled service is Bit-Integrity.	4
330	DURACLOUD-526	UI - Copy multiple content item across storage providers	This feature is to expose the copy-content-across-storage-providers functionality within Duradmin. This needs to include the option to "multi-select" content items to be copied as well as only selecting a single item. Likely this will be an addition to the existing content copy UI element. The interface should optionally allow the user to indicate the destination storage-provider.	2
331	DURACLOUD-527	Content manifest	This feature is to create and maintain a manifest (space-id, content-id, md5) of all content held in the primary storage provider. A listener should update the manifest whenever content is added or deleted from Durastore. The format of the manifest should be compatible with the expected input format of the bit-integrity service.	3
332	DURACLOUD-528	Migrate SCM to Git	This task is to migrate the DuraCloud SCM from Subversion to Git.	5
333	DURACLOUD-532	Rename spaces	This task is to make it possible for users to rename DuraCloud spaces. In order to do this, space names will need to no longer be directly tied to the bucket/container names of the underlying storage provider. One way to do this is for the name of a space to become a property of that space. A side benefit of this is that we should be able to remove many of the naming limitations we currently have on space names.	5
334	DURACLOUD-533	Archive-It Integration	This feature is to provide an integration with Archive-It so that users with Archive-It content may seamlessly push their collection into DuraCloud.	5
335	DURACLOUD-534	UI - Provide hierarchical content view	DurAdmin should be able to graphically display folder and file hierarchies in a similar fashion that is available on a local machine. Currently, the naming convention to indicate hierarchical files (ex. folder1/folder2/folder3/filename.txt) is not sufficient from a user perspective. It is expected that the interface will allow them to create "sub-folders" or spaces within spaces, etc. that will map to the local directories that the content was stored in. This task is to update the display of content items such that the user sees the commonly expected folder structure, which allows navigation down to files. This should look pretty similar to the display provided for content in S3 by the AWS console.	8
336	DURACLOUD-537	Long-running content actions	This task is to add support in DuraStore for long-running actions that can be run over content. Some examples of these kinds of tasks are: multi-file delete, multi-file copy (within and between providers), and multi-file property add/update/delete. It may make sense to implement this feature using the existing tasks infrastructure.	5
337	DURACLOUD-541	Integrate working version of CloudSync	This task is to pull in an updated version of CloudSync into the cloudsyncservice that does not have the issue of not being able to connect to a DuraCloud instance.	1
338	DURACLOUD-542	Upload tool web deployment	This feature is to establish a means of deploying the UploadTool via Duradmin as a signed-applet, webstart app, or other.	3
339	DURACLOUD-544	Automatically insert logo in x-duracloud-admin	When a DuraCloud instance is deployed, it should automatically include a logo in the x-duracloud-admin space, so that the logs don't get junked up. I have attached a possible logo to use for this purpose.	1
340	DURACLOUD-545	Implement an automated performance testing framework	Assemble a framework for automated performance testing to fit into the DuraCloud software engineering process. This framework is to include tests and test data sets. It will be implemented incrementally in an agile fashion. The primary goal of performance testing is to ensure that DuraCloud meets its goals for scalability and reliability. It characterizes the performance of the system under various load scenarios. It is also a primary input for bottleneck analysis that can be used to improve system performance. Performance tests often overlap functional tests and are useful for ensuring that DuraCloud is functioning as documented. Performance testing is also used to help size and manage the operating infrastructure in order to avoid unnecessary costs. These costs are part of the information needed to set the price of our services.	5
341	DURACLOUD-546	Update Sync Tool to take advantage of improvements due to the Upload Tool	The creation of the Upload Tool included several back-end changes to the Sync Tool. This task is to make improvements to the front-end of the Sync Tool to: - Display the progress of file transfers when printing status - Allow for individual files to be watched as well as directories	2
342	DURACLOUD-547	Upload tool applet cannot be used at the same time on spaces of the same name on different providers	If you are using the upload tool via DurAdmin to perform an upload on Amazon in space "test", you cannot also perform an upload to a space with the same name in another provider - you are sent to the existing upload window.	1
343	DURACLOUD-548	ACL: Create UI for managing ACL users/groups during space administration	This feature is to create a UI in duradmin for managing space-level ACLs. It will likely be a button in the righthand pane, similar to the "edit" button for space metadata. The functionality should enable Admins to add/remove 'users' and/or 'groups' to the space ACL with permissions of 'read' or 'write' (which implies read). A potentially attractive UI widget would be the one that renders two, side-by-side boxes containing usernames, and arrows allowing the moving of usernames from one box to the other.	3
344	DURACLOUD-549	ACL: Update StoreClient with setSpaceACL()	This feature should add the new method to the StoreClient: setSpaceACL() This method should take a spaceId and a map of 'usernames' and/or 'groupnames' and their associated privilege (read or write), and set the ACL for the given space.	3
345	DURACLOUD-550	ACL: Update DuraStore security to ACL rules	This feature should update the durastore spring security definitions to limit access as defined in: https://wiki.duraspace.org/display/DSPINT/Space+Level+ACLs+-+design	2
346	DURACLOUD-551	Space and content list reload	Please add a refresh button to the space listing as well as to the content listing within a space. Currently a space is not reloaded after content items are added, and a user must either (a) reload their entire browser window or (b) click on the space tab and renavigate to the appropriate space to see the newly added content items.	1
347	DURACLOUD-553	ACL: Update app-config and security-user bean to include 'group' details	This update is to modify app-config, SecurityUserBean, and security-users.xsd, as appropriate to support the inclusion of ACL 'groups' during instance initialization.	1
348	DURACLOUD-554	ACL: Add ACLStorageProvider	This feature is to add a new StorageProvider wrapper implementation to facilitate filtering the spaces listed in a getSpaces() request. The primary objectives are to: - provide a way to only return spaces that are either 'open' or are accessible to the calling user - improve performance so that each request for the space listing does not require querying the properties of each space from the underlying storage provider. Below are some implementation suggestions: 1. in StorageProviderFactoryImpl.getStorageProvider() * wrap storageProvider with a new ACLStorageProvider() 2. ACLStorageProvider * overwrite: getSpaces() * cache spaces and space acl properties	2
349	DURACLOUD-555	ACL: Enable multi-map content/space properties	This feature is to enable multi-map header values throughout the application for content and space properties. The capability should be enabled for the entire stack: storageproviders, to REST, to StoreClient, to Duradmin.	2
350	DURACLOUD-556	UI Copy Single Content Item Across Storage Providers	NULL	1
351	DURACLOUD-557	ACL: Update duradmin UI to exclude tabs from USER role	This update is to not display the "Services" tab nor the reports tabs under "Dashboard" if the user who is logged into duradmin only has ROLE_USER. Rather, only users with ROLE_ADMIN should be able to see the services and reports tabs.	1
352	DURACLOUD-558	ACL: Represent space access (open/closed) as an ACL	This improvement is to refactor the implementation of space access from the 'space-access' property to an ACL property (acl-group-public). In addition to cleaning up specialized logic for space access, handling space access as an ACL will allow a single call to the underlying storage providers to be made in determining access rights in Spring Security instead of a call for space access and a separate call for space ACLs.	4
353	DURACLOUD-560	UploadTool changes ACLs on upload	This bug is that when running the UploadTool, the existing ACLs of a space are replaced by a single ACL of {user=w}.	1
354	DURACLOUD-561	Deprecate space-access methods	This improvement is to deprecate the setSpaceAccess and getSpaceAccess methods on the StorageProvider and ContentStore interfaces.	1
355	DURACLOUD-563	Provide a REST call to allow administrators to retrieve a listing of all users and groups	In order to be able to create an ACL for a space, an administrative user needs to list all of the users and groups which should have access to the space. What makes this hard is that there's no way via the REST API for an administrator to get a list of the users and groups which are available. So while the ACL can be created currently, knowing how to fill it out is a challenge. This task is to add a call to the REST API which allows administrators to retrieve a listing of all users and groups, as well as which users are parts of which group.	3
356	DURACLOUD-565	Sort permissions display	This task is to sort the listing of items in the Permissions section of the content item display in DurAdmin. Groups should be displayed above users, and both sets should be sorted alphabetically (A-Z). The "public" group, when added, should always be displayed at the top of the list.	1
357	DURACLOUD-566	Handle space properties cache loading on instance initialization	Currently, the cache for space properties is loaded the first time a user attempts to access a storage provider. If this user is a non-admin user, and there are many spaces in the account, this action can take several minutes. This task is to transition to loading the cache when the instance is first initialized, rather than waiting for a user to access the instance, greatly lowering the likelihood that any user will have to wait while the cache is populated.	2
358	DURACLOUD-567	Show only writable spaces in the copy content dropdown (in the copy dialog)	NULL	1
359	DURACLOUD-568	Multi-space property editing: User should be alerted if they are trying to update one or more readonly spaces.	Suppose a user is looking at a list of four spaces and suppose he has write access to only three of them. Currently the set of editable tags and properties is gathered from all selected spaces regardless of whether the user has write access. Similarly if a user tries to add properties or tags to a selected space to which he has only read only access, the system won't handle it very beautifully. The ideal solution would be to disallow selection of unwritable spaces.	2
360	DURACLOUD-569	Create Messaging over setSpaceACLs()	This improvement is to add messaging over the durastore calls to StorageProvider.setSpaceACLs(). The configuration for this is in: durastore/src/main/webapp/WEB-INF/config/aop-config.xml A new advisor (ala SpaceUpdateAdvice) will needed to be created as well.	1
361	DURACLOUD-570	Copying content from Amazon to another provider produces unnecessary properties	When performing a content copy from Amazon to another provider, the resulting content item includes duplicate copies of "default" content properties, such as mimetype and checksum. This task is to filter out these properties when performing a copy.	2
362	DURACLOUD-573	Add "Help Desk" link to footer	Please add the text "Help Desk" to the footer before the "Contact Us" link. It should link to: http://duracloud.kayako.com.	1
363	DURACLOUD-574	Remove Duradmin's dependency on the Space Access APIs in storeclient	Remove the dependencies on space access by having duradmin use the acls instead. It is also necessary to make it clear to the user how to enable public access on a space if no public acl has been set. Additionally the public group should be highlighted in the UI to amplify its uniqueness.	2
364	DURACLOUD-575	Synctool: not re-syncing content deleted from DuraCloud	This issue is to fix the bug demonstrated in the following scenario: # run synctool pushing one or more files to a space ** observe success # delete one of the content items in DuraCloud newly pushed to the space # re-run the synctool ** observe deleted item is not re-uploaded The strategy to fix this issue is to introduce a new flag which instructs the sync tool to perform a "clean start" rather than a re-start, so that all files in all local directories are considered as part of the sync. This flag would essentially have the same effect as clearing the work directory prior to starting the sync tool. The flag should not be enabled by default.	1
365	DURACLOUD-577	/init resource should be available to ROLE_USER	This item is to address the bug that only anonymous users can inspect durastore, duraservice, and durareport. ROLE_USER and ROLE_ADMIN should also be able to make such an inspection.	1
366	DURACLOUD-578	Links into the djatoka viewer are broken in duradmin	Links into the djatoka viewer are broken in duradmin. For example the following url doesn't work: http://174.129.12.28/adore-djatoka-p18080/viewer.html?rft_id=http%3A%2F%2Fdanny.duracloud.org%3A80%2Fdurastore%2Fimage-transformer-dest%2FSunset.png%3FstoreID%3D0 But this one does: http://174.129.12.28/adore-djatoka-p18080/viewer.html?rft_id=https%3A%2F%2Fdanny.duracloud.org%2Fdurastore%2Fimage-transformer-dest%2FSunset.png%3FstoreID%3D0 The difference between the above urls: replace the durastore url protocol with https and remove the port and everything works.	1
367	DURACLOUD-579	Content image does not refresh after clicking "Make space public" button.	To reproduce: 1. Make sure image service is running. 2. In duradmin browse to an image content item in a non-public space. 3. Click the make space public button. 4. Notice: the image is not visible. 5. Also note that the warning message is not automatically removed as it should be. Also note that when making a space non-public, the button for making it public does not reappear. I think the panel should be refreshed and the image should be immediately available.	1
368	DURACLOUD-581	January 2012 completed reports are not showing up in duradmin dashboard.	To reproduce, run a service to completion and then observe in the duradmin dashboard that it as well as another other service run after January 1st is not being displayed in the dashboard.	1
369	DURACLOUD-582	Sync tool: Create a file transfer log	This task is to update the sync tool such that it creates and updates a textual transfer log including all files which are added or updated in DuraCloud along with their checksum. This file would only include files which were successfully transferred to DuraCloud. Ideally the format of this file would follow the format expected by the bit integrity service, so that after doing a fresh transfer of a full dataset into DuraCloud this transfer log could be compared to the space via the bit integrity checker service.	2
370	DURACLOUD-583	Verify that pointing multiple subdomains to a single instance does not cause issues	This task is to perform tests against a DuraCloud instance where multiple subdomains are pointed. One subdomain will be used to initialize the instance, while others will be used to access the UI and APIs. This is intended to simulate the interaction that users will have with the Community Cloud. Ideally, coming to DuraCloud through each of the different host values will show that: 1. The DuraCloud applications function correctly 2. The user is always shown only their host value via the UI. Note that the user interaction can be limited to User role actions.	1
371	DURACLOUD-584	Ensure that client tools continue to function across minor outages	This task is to test the DuraCloud client tools (primarily Sync Tool and Retrieval Tool) and make updates that are necessary to allow the tools to continue to run even if the DuraCloud instance becomes unavailable for a short period of time. The period of time to be tested against should be the time that is required to upgrade a DuraCloud instance to a new version. This will allow DuraCloud users to keep their tools running even across an upgrade cycle. A likely strategy is: When a transfer fails, check to see if the instance is available before doing a retry, if its not, then wait for ~5 min and try again. When the instance becomes available again, resume file transfers.	2
372	DURACLOUD-585	Distinguish service-failures from failed-service-results for Non-Hadoop Services	Same as https://jira.duraspace.org/browse/DURACLOUD-512 but for non-hadoop services that operate on lists of items.	2
373	DURACLOUD-586	When editing multiple space properties, it is possible for users to attempt to update read only spaces.	To reproduce this problem: 1. select a writeable spaceA and read-only spaceB (using the checkboxes) 2. click edit properties 3. add a tag or name value pair and save. 4. duradmin displays the error dialog (unauthorized action error)	1
374	DURACLOUD-587	When logged in as a non-admin, the multi-space delete button should not be visible.	To reproduce: 1. log in as a user 2. select a space or two using the checkboxes in the spaces list. 3. confirm that a delete button is visible.	1
375	DURACLOUD-588	Verify the SDSC storage provider integration	This task is to ensure that the SDSC storage provider works as expected, that content can be stored and retrieved, and that everything looks as it should on the SDSC side. Tests should be conducted to verify that all DuraCloud APIs work as expected when executed against the SDSC provider. Tests should also be conducted to verify that the SDSC provider can handle storing and retrieving large amounts of content. This task requires both continuing the conversation with the SDSC folks as well as direct testing and validation.	3
376	DURACLOUD-589	Set expiration policies for log spaces	This task is to use Amazon's management console for each of the current DuraCloud accounts to set an expiration policy on the x-service-work space to remove all content items with the "logs" prefix after 30 days.	1
377	DURACLOUD-590	Create and test 64-bit AMI	This task is to: 1. Test the DuraCloud software on a 64-bit instance, to ensure that it works properly, and fix any problems that are discovered 2. Create a production 64-bit AMI as part of the release cycle	2
378	DURACLOUD-591	Various DurAdmin updates	Update the following items in DurAdmin: - Remove the option to "add one item", change the "add many items" link to simply be "add items" - Update the image on the copy button to appear consistent with the other buttons (currently appears to be lower quality) - In the Content Detail panel, make content size number understandable (at least include a unit - "bytes") - In the Content Detail panel, set all buttons to be the same color (light gray perhaps)	1
379	DURACLOUD-592	DurAdmin updates	Update the following items in DurAdmin: - Change "Space ID" and "Content ID" to "Space Name" and "Content Name" respectively - In the footer of every DurAdmin instance page, ensure that DuraCloud has the "c" capitalized. - In the footer of every DurAdmin instance page, the link to DuraCloud.org should come before DuraSpace.org - In the footer of every DurAdmin instance page, replace the current "Help Desk" link with a "Help Center" link which points to: https://wiki.duraspace.org/display/DURACLOUD/DuraCloud+Help+Center - The title of every DurAdmin instance page (that appears in the browser tab) should be DuraCloud: Login, DuraCloud: Spaces, etc. or something similar (not "DurAdmin :: DurAdmin: Login" or "DurAdmin: Spaces") - Change "DuraCloud Administrator" to just "DuraCloud" - Make the Dashboard tab available only to root users	1
380	DURACLOUD-593	DurAdmin: restrict access to Administration tab	This task is to limit the Administration tab in DurAdmin to be viewable only by root users.	1
381	DURACLOUD-594	Move storage reports onto spaces tab	This task is to break out the information in the storage reports and display it instead on the main Spaces tab. The space-specific charts should be included on the Space Detail page. The provider-specific charts should be displayed in the detail area that is currently shown as blank when selecting the Spaces tab. The top-level all-providers charts will no longer be displayed. Both the provider-specific and space-specific charts should still provide the option to view historical charts and data.	5
382	DURACLOUD-598	when uploading jpg files with hashmark in file name, the mime-type is not accurately identified.	To reproduce: 1. Run the bulk uploader in duradmin. 2. Select a jpg with a # mark in the filename such as test#1.jpg. 3. Select a jpg with without a # mark in the filename such as test2.jpg. 4. Upload files. 5. Refresh space in duradmin 6. Verify that the mimetype for test2.jpg is image/jpeg (correct) 7. Verify test#1.jpg is application/octet-stream (less correct)	1
383	DURACLOUD-599	When uploading with bulk upload tool, if file is a jpg with hashmark in file name, the mime-type is not accurately identified.	To reproduce: 1. Run the bulk uploader in duradmin. 2. Select a jpg with a # mark in the filename such as test#1.jpg. 3. Select a jpg with without a # mark in the filename such as test2.jpg. 4. Upload files. 5. Refresh space in duradmin 6. Verify that the mimetype for test2.jpg is image/jpeg (correct) 7. Verify test#1.jpg is application/octet-stream (less correct)	1
384	DURACLOUD-600	Release 1.3.1	This task is to run the attached space-acl-tool over all production accounts to convert the space-access properties to space ACLs. Note that when running the utility in "CONVERT" mode, it simply creates the ACLs based on the space-access property. If you want to remove the legacy space-access properties, the utility must be run in "REMOVE" mode.	1
385	DURACLOUD-602	In the copy item dialog, there is no validation for missing destination space info.	To reproduce: 1. as admin, remove write access to all spaces within storageProviderA. 2. as user, login, select a content item, click copy, and select storageProviderA. 3. Verify that the destination space drop down is empty. 4. click ok. 5. verify that the call is made despite the invalid data. Optimal solution: add default list item to the space select ("Select a destination space") and display invalid form message when this item is selected.	1
386	DURACLOUD-603	Make the image thumbnails in duradmin larger and improve the layout.	When viewing an image in a public space with the image server service running, make the thumbnail larger and center the layout.	1
387	DURACLOUD-604	Provide visual feedback for slow loading thumbnails from the image server	Image thumbnails are generated on the fly and then cached by the djatoka image server. The latency in thumbnail generation can sometimes leave the user wondering what happened to the image. Providing a visual cue to show that something is happening, albeit slowly, would improve the experience.	1
388	DURACLOUD-605	Retrieval Tool: Space error feedback	This improvement is to add user feedback in the case where the space(s) being retrieved is inaccessible. When a space does not exist or when the user does not have access to the target space, the retrievaltool output message is as follows: ========================================= Retrieval Tool processing complete, final status: -------------------------------------- Retrieval Tool 1.4.0-SNAPSHOT - Status -------------------------------------- Start Time: 2012-01-20 09:34:57.106 Current Time: 2012-01-20 09:35:17.108 Retrievals In Process: 0 Successful Retrievals: 0 No Change Needed: 0 Failed Retrievals: 0 -------------------------------------- ========================================= In order to help the user understand why the retrieval did not work, a more descriptive status message would be helpful.	1
389	DURACLOUD-608	UI issues with prefix + refresh	Part 1: Following these steps: 1. Select a space which has more than 1000 items 2. Enter a prefix into the "type prefix" box, hit enter 3. Click the refresh button on the content item listing Causes a series of error dialogs to pop up, indicating a NullPointerException while DurAdmin is attempting to count the space size. The stack trace reported in the dialog is captured in the attached file. Part 2: Following the same steps as above, but selecting a space with less than 1000 items causes the "Space Detail" area of the screen to blink continually	1
390	DURACLOUD-611	Service Report Link in Service Details	This bug can be seen when running a service that creates an output report (such as bit-integrity-tools: generate integrity details for space) in a storage provider other than the primary. For example, run the above service over a Rackspace space. Once the service is complete, select the link in the service details to show the results. Notice the report is not found due to the missing '?storeID=X' param.	1
391	DURACLOUD-612	Create HP Storage Provider	This task is to create an HP Storage Provider. HP is still in its testing phase, so this will allow us to test the new provider more easily, and determine the extent to which it meets the expectations of the DuraCloud application. HP uses OpenStack, so the code required for this implementation should be minimal.	1
392	DURACLOUD-615	Integrate with Shibboleth-based systems	This task is to make it possible for Duradmin to integrate with existing Shibboleth-based authentication and authorization systems. This is needed in order to satisfy the requirements of Internet2 and DfR. See discussions related to this issue here: https://wiki.duraspace.org/display/DFR/Shibboleth+Thread	3
393	DURACLOUD-616	Space and content checkboxes are hard to click on	When trying to select a set of spaces to work on, I often select several spaces and then accidentally deselect them all accidentally by clicking a few pixels off of another checkbox. The solution here is to add padding to the checkbox element so that it is not necessary to be so precise with the mouse position.	1
394	DURACLOUD-617	Eventual Consistency on Secondary StorageProviders: CreateSpace	This improvement is to increase the probability that the space creation method will work on secondary providers. Currently, the S3StorageProvider has a wait cycle in the createSpace() method that retries adding the space-metadata file until the bucket has been created and registered. A similar wait cycle should be added to the createSpace() method (or rather appropriate 'setSpaceProperties' method) of: - RackspaceStorageProvider - AzureStorageProvider	1
395	DURACLOUD-618	Add Space Dialog	This task is to address the following bug: When selecting "Add Space", the flyout allows for entering the new space name. When hitting the "enter" key instead of the "Add" button, a pop-up with the following error is displayed: error: space is undefined	1
396	DURACLOUD-619	Exception on Space Delete	This task is to fix the bug that causes an exception to be thrown when deleting a space either in duradmin or via the REST api. With any storage-provider, when deleting a space in duradmin, a pop-up with the following message is displayed: failed to delete space! In looking at the tomcat log, an error is also shown when using the REST api. In both cases, however, the space is actually deleted.	1
397	DURACLOUD-620	Duplicate on Change at Space-Level	This improvement is to update the Duplicate on Change service to allow configuration at the space-level. Namely, the service should be able to run on a subset of the total spaces of an account. It would be ideal if this improvement also allowed the spaces over which Dup-on-Change is configured to copy from the primary to any of the secondary providers.	5
398	DURACLOUD-621	Display bit integrity results on each space	Services are being automated and users are no longer required to run the bit integrity service themselves, but they still need to know that their content is being verified. This task is to include on the UI, likely in the space details area, an indication of when the most recent successful bit integrity check was run over the content in that space. In the event that the most recent bit integrity check resulted in an error, we (DuraCloud staff) will need the opportunity to investigate the issue and correct it if possible before the issue is displayed to the user (this may also be done in an automated fashion.) If it turns out that the error requires user intervention, then it would be nice to be able to have that be indicated in some way on the UI, calling out specifically which files need to be addressed.	2
399	DURACLOUD-622	Refactor RackspaceStorageProvider to OpenStackStorageProvider	This improvement is to refactor the RackspaceStorageProvider to OpenStackStorageProvider. Since we currently have three storage-providers that all expose an OpenStack API, the RackspaceStorageProvider (which is coded to the OpenStack API) should now be abstracted up as the base class of the three implementations: Rackspace, SDSC, and HP. Each of these inheriting classes should have their own names displayed in error and logging messages.	1
400	DURACLOUD-624	Streaming URL for spaces streamed by the Media Streamer	Release 2.0 added the ability for users to select a button on a space and turn on media streaming for that space. This task is to ensure that the streaming URL provided by the Media Streamer service is added to the metadata of the space when it is being streamed, so that the user has access to it.	1
401	DURACLOUD-626	Media Streamer: remove Viewer space	Since services will no longer be run directly by users, there is no longer a need for the Media Streamer service to produce the contents of the Viewer space. Those generated files are not used as part of the service, they are only provided as examples for how a user might integrate the media streamer service with their own website. This task is to: - Add a copy of the generated files (updated to be more generic) with some surrounding instructions to the DuraCloud documentation - Update the Media Streamer service to no longer generate these example files	1
402	DURACLOUD-627	Sync and Retrieval Tool encryption support	After encryption support is added to the StoreClient, the Sync and Retrieval tools need to be updated to allow the user to choose to enable content encryption and to supply the necessary encryption key.	2
403	DURACLOUD-628	Add user-id to content messaging	This improvement is to include the authenticated user-id of the user performing content actions over the durastore API. At the least, the user-id needs to be included on the ingest event message. This is in support of the DfR: Object Creation Service.	1
404	DURACLOUD-629	Update Media Streamer Service to not Require Sources	This task is to remove the requirement of the media streamer service to have a source space.	1
405	DURACLOUD-630	show content manifest head in space detail with link to full manifest	The purpose of this feature is to guide the user to the content manifest for the space as well as enable them to peek at the recent activity. The "peek" might be equivalent to head -10 audit-log.	2
406	DURACLOUD-633	Snippet Generator for Streaming Content	It would be nice to have way to generate a snippet for streaming audio/video that can be embedded directly into any page.	2
407	DURACLOUD-634	Formatting on "upload-tool" space	The display on the Spaces tab causes any space named "upload-tool" to be formatted incorrectly. This is because the <div id="upload-tool"> that is created for the space on the page happens to match up with a css entry for #upload-tool (which is meant for a div in upload-tool.jsp). As it turns out, there are several space names which can cause odd UI behavior for the same reason, such as: "spaces-list-view-resizer", "footer-content", "logo-ds" (which is particularly strange looking), etc. There should be a strategy for making sure that these kinds of name collisions don't happen. The attached file shows how the display reacts to the "upload-tool" space.	1
408	DURACLOUD-635	Media Streamer on/off button in the UI	NULL	2
409	DURACLOUD-636	Executor Client	This task is to create a java client (similar to storeclient, serviceclient, and reportclient) to simplify interactions with the Executor REST API.	1
410	DURACLOUD-637	Add manifest client	This task is to create a Java client for calling the manifest REST methods in DuraBoss.	1
411	DURACLOUD-638	Consolidate DuraBoss Interfaces	This improvement is to refactor the Executor, ManifestGenerator, and Auditor interfaces to a common project that can be shared between DuraBoss and the various DuraBoss clients. Additionally, these interfaces should be implemented by the utilities themselves as well as the clients.	1
412	DURACLOUD-639	Make charts zoomable	Space pie charts should be zoomable, displaying a larger graph in an overlay.	1
413	DURACLOUD-640	Make Spaces pie charts navigable	Space slices in the cross spaces pie charts should be clickable. On click, the user should navigate to the same time slice but on the space detail history component.	2
414	DURACLOUD-643	Executor: Media Streaming	This task is to implement the Media Streaming Handler in the Executor. Design notes can be found here: https://wiki.duraspace.org/display/DSPINT/DuraBoss+Design	3
415	DURACLOUD-644	Executor: Bit Integrity Checking	This task is to complete the Bit Integrity Handler in the Executor. Design notes can be found here: https://wiki.duraspace.org/display/DSPINT/DuraBoss+Design	10
416	DURACLOUD-646	Create Auditor Client	This task is to create a java client (similar to storeclient, serviceclient, and executorclient) to simplify interactions with the Auditor REST API.	1
417	DURACLOUD-647	SystemUserCredential Across Webapps	This improvement is to create the SystemUserCredential in a similar was to how RootUserCredential is created, via a default username/password over-written by system properties. Once this is done, DuradminExecutorImpl.java should be evaluated for exchanging root-user with system-user.	1
418	DURACLOUD-648	Merge common-notification into notification project	This task is to move the code which currently exists in the common-notificaton project into the notification project. An update should be made to inject AmazonNotificationFactory into EmailNotifier in DuraBoss, so no dependency on notification-amazon is required in the notification project. As part of this transition, the implementation classes should be pushed into the package org.duracloud.notification.impl.	1
419	DURACLOUD-650	Newly uploaded content items do not show up on clicking the space.	To reproduce: 1. Open a space. 2. Upload content into a space. 3. Click on the space again in the space list. 4. Notice the newly uploaded content is not showing up.	1
420	DURACLOUD-652	Add multiple tags at once to spaces and content	It should be possible to enter multiple tags at once. A tag should not contain spaces. Let us modify the UI so that you can enter a list of space or comma separated words that get entered as separate tags.	1
421	DURACLOUD-653	Tag and Metadata deletes are not being serialized	Tag and Metadata deletes are not being serialized. If one deletes multiple tags/key value pairs in rapid succession, you'll notice that some items that were deleted reappear upon refresh. Solution: use deferreds to serialize calls.	2
422	DURACLOUD-655	Deselecting all content items after having selected one or more does not cause the detail pane to show the space detail as it should.	To reproduce: 1. select a content item using the checkbox. 2. deselect it. 3. notice the space detail pane is not being displayed.	1
423	DURACLOUD-656	The checkbox used to select all items in the content items list should be cleared when selecting another space	1. select a space. 2. select all content items. 3. select another space. 4. notice the "select all" check box is still checked.	1
424	DURACLOUD-657	Refactor revision 914 so that it is not necessary to resolve the primary content store id on every call to get space and content detail info	Probable solution will involve caching the primary content store on the client.	1
425	DURACLOUD-658	Manifest for empty space	Currently, when a new space is created, then a call to ask for the manifest is made, the response is a 404. Instead, the response should be a 200 and an empty manifest list should be returned. This could be resolved by creating the space audit log on space creation, rather than on the addition of the first content item.	1
426	DURACLOUD-660	Chart Labels are off on panel resize	To reproduce: 1. Display a space 2. Make the Space Detail pane wider. 3. Note how the line graph labels are skewed.	1
427	DURACLOUD-663	Secondary Provider Audit Entry Sequencing	This issue is to resolve the bug occasionally seen in the content manifest when running dup-on-change. Sometimes ingest and delete messaging events lose sequence on the secondary provider. To create this scenario, an item must already exist in a space, then be deleted and immediately re-ingested. It is not a common production scenario. The attached audit log demonstrates such a case. Note, the actual content in both the primary and secondary stores were actually the same, but the secondary store does not show file-6.txt in its manifest: space-id content-id MD5 manitest file-0.txt b1946ac92492d2347c6235b4d2611184 manitest file-10.txt b1946ac92492d2347c6235b4d2611184 manitest file-1.txt b1946ac92492d2347c6235b4d2611184 manitest file-2.txt b1946ac92492d2347c6235b4d2611184 manitest file-3.txt b1946ac92492d2347c6235b4d2611184 manitest file-4.txt b1946ac92492d2347c6235b4d2611184 manitest file-5.txt b1946ac92492d2347c6235b4d2611184 manitest file-6.txt b1946ac92492d2347c6235b4d2611184 manitest file-9.txt b1946ac92492d2347c6235b4d2611184	2
428	DURACLOUD-665	Exceptions being swallowed by javascript on space retrieval in Duradmin	See line 1080 of spaces-manager.js Notice only a 404 error is being handled.	1
429	DURACLOUD-666	Enable Duradmin to run separately on its own instance.	See attached error: There appears to a problem calling remotely into duraservice via duradmin.	2
430	DURACLOUD-667	Storage pie charts in duradmin/spaces should display file size and count totals	Add	1
431	DURACLOUD-668	Update DuraBoss security-config.xml	This task is to update the security-config.xml file in DuraBoss to appropriately cover all parts of the DuraBoss REST API	1
432	DURACLOUD-669	Update to the new Service Plans	Once the new set of service plans have been nailed down, this task is to make any updates that are necessary to make new plans available and remove unneeded plans.	1
433	DURACLOUD-672	Logback Memory Leak	This improvement is to determine a way to allow logback in the DuraCloud webapps to - monitor for logback.xml updates without requiring application server cycling, and - not create memory leaks The memory leak can be seen by re-deploying the DuraCloud webapps and noticing that there are warnings in catalina.out and that the old processes are not destroyed.	2
434	DURACLOUD-673	SDSC as Primary Storage	This task is to determine the feasibility of using SDSC as a DuraCloud primary store. Beyond function feasibility, the performance differences of an EC2 to SDSC configuration should be compared to the standard EC2 to S3 configuration. As a side note, it would be interesting to explore SDSC's openness to supporting "salted" server-side checksums.	3
435	DURACLOUD-674	Chronopolis REST Server Client	This feature is to create a java client for the Chronopolis REST Server. https://github.com/msmorul/chron-notifier Further details of the usage of the Chronopolis client can be found here: https://wiki.duraspace.org/display/CHRONO/DuraCloud+Chronopolis+Integration	3
436	DURACLOUD-675	Chronopolis Utility Integration	This feature is to leverage the Chronopolis REST client to integrate the business tier of requesting: - backups to Chronopolis - restores from Chronpolis - notifications on errors and completions - polling of in-process requests	5
437	DURACLOUD-676	Chronopolis UI Updates	This feature is to integrate the ability to request backups and restores to/from Chronopolis into Duradmin.	2
438	DURACLOUD-678	Do not display blank "Size" property for spaces	The "Size" property for spaces (the number of bytes stored in that space) is always available for spaces in Rackspace, because this information is provided, but is never available for spaces in Amazon and some other providers. This task is to remove the "Size" property from the space display when it is empty.	1
439	DURACLOUD-680	Buttons in Space and Content detail should wrap	Currently, when viewing DurAdmin on a small screen or window, buttons in the Space Detail and Content Detail areas can be unavailable. Since there is no horizontal scroll bar, it's not obvious that those additional buttons even exist. Looking at the attached screen shot, it is not obvious that there are View and Delete buttons which do not show up on the screen. This task is to allow these buttons to wrap, so that they will remain available even as the screen size diminishes.	1
440	DURACLOUD-685	Label Storage Graph Axes	This improvement is to add labels to the two y-axes of the "History" "Byte and File Counts Over Time" storage graphs. Currently it is not exactly clear that the left-hand y-axis represents cumulative file size, and that the right-hand y-axis represents cumulative file counts.	1
441	DURACLOUD-686	Item-level Bit-Integrity Timestamps	This improvement is to record the timestamp at the item-level for bit-integrity runs. This would allow for providing the "green bar" of bit-integrity health for each item. Likely, the timestamp would be persisted during the bit-integrity runs in each content line-item in the service-generated report.	1
442	DURACLOUD-688	Bagit Manifest: Two Spaces	This improvement is to make sure that the ManifestGenerator for Bagit manifests delimits the checksum from the path with two spaces instead of the current single space character.	1
443	DURACLOUD-689	Top-level Default Error Page	This improvement is to render a default error page at the apache level. For example, https://subdomain.duracloud.org/junk should display a more user-friendly page.	1
444	DURACLOUD-694	DuraCloud 2.0 UI doesn't load when using Opera or Internet Explorer web browsers	When accessing DuraCloud from Opera or IE, the UI doesn't load properly and is "unusable" (i.e. you cannot do anything). Both browsers provide the same result. Here's a summary of what happens: 1. Visit http://[your-site].duracloud.org/ using the Opera or IE 2. Login -- this works fine 3. You are presented with a screen which shows no Spaces or content items. (I.e. it looks as though all your content has disappeared). At this point, there's nothing you can do except logout. This occurs on both Mac OSX and Windows 7 when using Opera, and on Windows 7 when using IE.	1
445	DURACLOUD-695	Bulk Upload Applet Doesn't Work or Hangs on Multiple Browsers	From the last All Staff Retreat, I know this is a "known" issue. But, I wanted to track the scenarios that I've discovered thus far. The Bulk Upload Applet responds in the following ways when loaded in the following environments: * Mac OSX (10.6.8) + Firefox 11.0 - Initially, saw a "permanent" hang. I had to kill Firefox to continue. However, after restarting Firefox, on second attempt it loaded almost immediately. * Mac OSX + Chrome 18.0 - Temporary hang (1-2 mins), eventually loads properly. * Mac OSX + Safari 5.1.5 - Loads almost immediately. * Windows 7 + Chrome 18.0 - Loads in about 20-30 seconds * Windows 7 + Firefox 12.0 - Applet never seems to load & I always see a "blank" pop-up window (just includes Duracloud header/footer) UPDATE: Actually the issue was that the Java Plugin was disabled. See first comment below. Just as a note, Internet Explorer & Opera also likely should be tested once DURACLOUD-694 issue is resolved.	1
446	DURACLOUD-696	Pie charts not visible on history line graph data point click	When an admin is logged in and looking at the space detail, the pie charts will display below the fold. In order to see them, the user scrolls down. However when the user clicks on a line chart data point, the pie graphs will be updated, but the user will not notice because they are not visible. It would be much better if the history line chart would scroll to the top of its container when the line chart is clicked.	1
447	DURACLOUD-697	Allow admin to change custom logo by dragging and dropping it into place.	The current method of replacing /x-duracloud-admin/logo is not so user friendly.	2
448	DURACLOUD-698	back and forward browser buttons do not work in the spaces manager in IE 9	To reproduce 1) click a couple of spaces. 2) then uses the browser back button. 3) notice it did not go back as expected.	1
449	DURACLOUD-701	Space Names with '--'	Currently, when using either duradmin or the durastore REST API directly, creating a space which includes '--' behaves unexpectedly. For example, create a space named: 'test--x--space' It shows up in the space listing as expected. Then click the spaces tab, and it turns into: 'test-x-space' Likewise, when creating similarly named space via the durastore REST API. In both cases space shows up in the listing as well as in the AWS S3 Console as having a single '-'. It would appear as if the transformation of '--' into '-' is happening at the durastore level. This task is to allow durastore to permit space names with '--'.	1
450	DURACLOUD-702	UI: Space count causes bit integrity notice to be removed	On spaces with a large number of content items, there is an ajax call made to perform the count of items in the background. This number is updated in the space properties as the count continues, and is eventually set to indicate the final number. When a space has had a successful bit integrity run, a notice with a green background is displayed in the space properties, with a link to the results of the bit integrity run. In situations where both of these occur (a large number of items, and a successful bit integrity run), the completion of the item count causes the space properties to be reloaded, which removes the notice about the bit integrity run. This task is to ensure that the bit integrity notice persists through the item count update.	1
451	DURACLOUD-703	Delete space button available to user	When logged in with "user" role, space delete button is available/visible. After clicking to delete space, an error pop-up window appears that states "space delete failed." Ideally, a person logged in with "user" role would be unable to delete the space, but would have an easy way to delete ALL the contents stored within a space. See attached screenshot for error pop-up window.	1
452	DURACLOUD-704	Bit Integrity Tools service: Create report on empty space	Currently the Bit Integrity Tools service, when run over a space with 0 content items, indicates a link to a report, but does not actually generate the report. This task is to ensure that a report is generated, even when no content items are in the space.	1
453	DURACLOUD-706	UI: Remove image preview	Currently the image preview section of the UI displays an icon (with mountains) when either the Image Server is not on, or when the space is not publicly viewable. This tends to prompt lots of user questions around why they cannot view an preview. This is especially an issue for multi-tenant accounts, where the user is not able to start the Image Server or make their space publicly viewable. This task is to remove the image preview area of the content item display.	1
454	DURACLOUD-707	Secondary storage providers should be Read-only via the Rest Api	As the synchronization of content between storage providers becomes automated (DURACLOUD-642), it becomes unnecessary for users to edit content in their secondary provider. In order to maintain an assurance that content remains consistent between providers, the ability for users to make changes at their secondary storage provider should be removed. They would still be able to view and download all content, but no add/update/delete operations would be allowed. This task is to update the REST API to ensure that users with role user/admin/owner do not have write access to secondary stores.	3
455	DURACLOUD-708	500 on Image Content Selection	This bug can be seen when quickly clicking between content items with image mimetypes ('image/*') within duradmin. A 500 error is presented to the user.	1
456	DURACLOUD-710	UI: Space counting issues	This item covers two issues with the space count that occurs on the space properties of spaces with more than 1000 items: 1. The count now often starts at 0+. The count used to consistently start at 1000+ or 1999+. 2. When the count completes, it very often restarts and performs the count a second time, always ending with the same result.	1
457	DURACLOUD-712	SyncTool: Better handle periods of time when the DuraCloud instance is unavailable	Update the sync tool to be better able to handle periods of time where the instance has become unavailable. This is particularly important to allow the SyncTool to continue to run normally across DuraCloud instance upgrades. A likely strategy is: When a transfer fails, check to see if the instance is available before doing a retry, if its not, then wait for ~5 min and try again. When the instance becomes available again, resume file transfers.	2
458	DURACLOUD-714	UI: Display of content items names which include #. characters	If a content item with the name "#.txt" exists in a space, then when it is selected in the UI, the content detail will display for a moment, but will then act as if the spaces tab has been selected. It appears that both the hash and period characters need to be in the content ID, but other characters can be added and the behavior remains the same.	1
459	DURACLOUD-715	SDSC Auth Endpoint	This improvement is to switch the authUrl found in SDSCStorageProvider from: "https://cloud.sdsc.edu/auth/v1.0 to: https://duracloud.auth.cloud.sdsc.edu/auth/v1.0	1
460	DURACLOUD-716	SyncTool --dryrun	This improvement is to create a "dryrun" flag on the synctool. This flag would tell the synctool to go through the steps of determining which files should be sync-ed up to DuraCloud (including deleted and modified content), but instead of actually performing the sync, just a report will be produced detailing which content would have been sync-ed.	3
461	DURACLOUD-717	Synctool - Java7: SSLProtocolException	With Java7, the following exception is thrown when trying to use the Synctool in any standard configuration: Exception in thread "main" java.lang.RuntimeException: Could not create connection to DuraStore due to Error retrieving content stores. handshake alert: unrecognized_name The issue and workaround are posted here: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7127374	2
462	DURACLOUD-720	Use /mnt on EC2 Instances	This improvement is to leverage /mnt on EC2 instances for temp files, at the least. /var/log/tomcat6 should link to /mnt An analysis should also be performed to determine if /home or the webapps should use /mnt as well.	1
463	DURACLOUD-721	Limit Tomcat logs	This task is to update the log management settings for the Tomcat server on DuraCloud AMIs. These settings are contained in /etc/logrotate.d/tomcat6. Log rotation settings should be updated to rotate on size (perhaps 256M) rather than date, and limit the number of rotations which are kept (to perhaps 3). A full list of options for these settings can be found with "man logrotate".	1
464	DURACLOUD-723	Sync and Retrieval Tool metadata support	This task is to update the Sync Tool and Retrieval Tool to be able to handle content tags and metadata. A DuraCloud space should be able to be pulled down to the local file system using the Retrieval Tool and pushed back up to a new DuraCloud space using the Sync Tool without suffering any loss of metadata.	3
465	DURACLOUD-724	Initializing an instance to use locally stored services is not working	Performing an instance initialization by pointing to the local service repo (on the same host) no longer works. The issue is that the Executor in DuraBoss attempts to make calls to DuraService during initialization, but gets a 401 (unauthorized) error which halts the initialization process.	1
466	DURACLOUD-725	Audit Log on Space Delete	This issue is that the audit log (i.e. x-duracloud-admin / audit/storage-audit-log-<space>-<time>.tsv) does not detect deleted content when an entire space is deleted. Scenario: # Create a space # Add content to the space # Check the content manifest ** curl -u user:pass https://host.duracloud.org/duraboss/manifest/<space>?storeID=X ** note, the manifest is only written every 60seconds, so you may need to wait before the content is contained in the manifest # Delete a content item, and recheck the manifest ** the manifest should not show the deleted item # Delete the space # Check the content manifest ** BUG: the manifest still shows the items that were in the space prior to the space deletion	1
467	DURACLOUD-726	Auditor not passing tests on Windows	The Auditor project produces many errors and test failures when being built using "mvn clean install" on Windows. This is due to the use of colon characters in created file names (which are date based), and file path separator issues.	1
468	DURACLOUD-727	Auditor: Log re-creation on large spaces is failing	The auditor action to re-create the audit logs for all spaces is failing when being run over large sets of content. This has been seen on several instances where content within a space pushes into the hundreds of thousands of items. One suggest reason for this behavior is that the thread being used to create the log may run out of memory and exit.	2
469	DURACLOUD-728	Move DefaultFilter from duradmin to common.	The motivation for this refactoring is to make the aforementioned useful class available to other web applications such as dfr-sync and mc.	1
470	DURACLOUD-729	Double approval for content item deletes	When a user chooses to delete a content item in the UI, a pop-up window should confirm the action and then re-confirm the action (indicating that the secondary copy will also be deleted because of this action). Ideally, the double warning will protect from some "accidental" deletions of content through the UI. This will not extend to the sync tool or the REST API.	2
471	DURACLOUD-730	Audit log not updated: SDSC Restore	This issue is that when Chronopolis restores content into an SDSC container, the Audit log is not updated. This is due to the fact that back-dooring the content ingest by-passes Durastore and the messaging that the Auditor depends on.	2
472	DURACLOUD-731	Duplicaton on Demand testing	Several runs of the Duplication on Demand service have failed while attempting to transfer a large content set (500,000 files) from S3 to SDSC, for Chronopolis testing. This task is to review the failures, and perform further tests to determine the cause of the issue.	2
473	DURACLOUD-732	Duplicate-on-Demand via Inspector/Executor flow	As noted in testing described here https://wiki.duraspace.org/x/8YzvAQ, the work of transferring very large numbers of files from one storage provider to another is not particularly well suited to hadoop. This task it to create an alternative content duplication method. The requirements are as follows: - Must be able to move content from any provider to any other provider - Must be able to be deployed onto multiple server instances in order to share the load and speed up processing It may be necessary for this to be a multi-step process. Perhaps something like: 1. Run a process with the source space and the number of instances that will be used for the job as parameters, and it splits up the work into an input file for each worker node. 2. Each worker node is spun up and initialized with the proper information to do the work, including their portion of the job, as defined in step 1. 3. After all nodes have completed step 2, a final process verifies the transfer by comparing the source and destination spaces. The goals of this work: 1. Provide a method which can reliably transfer large numbers of content files from one provider to another. 2. Provide a starting point for the further work of developing a large file processing capability within DuraCloud that does not rely on Amazon EMR.	5
474	DURACLOUD-733	Sync Tool: CPU Utilization	The Sync Tool, when run, tends to use a very large percentage of the CPU. As noted by SaaS4, running the Sync Tool continuously for 10 days kept the CPU on their server pegged at 100% the entire time. It seems likely that the high CPU utilization is due to the constant generation of MD5 checksums for each file. This would be a particular issue when the file set contains a large number of very small files. This task is to investigate the reason behind the heavy CPU usage. Ideally a fix could also be included.	2
475	DURACLOUD-734	Sync Tool: Limit local logging	The Sync Tool currently generates three logs: complete.log, duracloud.log, and synctool.log. It was noted by the SaaS4 customer that these logs together took up an unnecessarily large amount of hard drive space. They noted that over 10 days, after pushing 500 Gb of data, the logs totaled around 300 Gb. This is clearly too much space consumed by logs. This task is to limit the log outputs. This could include removing the complete.log, as well as ensuring that logs are cycled properly, so as to not consume more than a set amount of space.	2
476	DURACLOUD-735	Sync Tool: Exclude list	This task is to allow for passing a list of files to the Sync Tool which should be ignored when performing a sync. The simple case is for the file to include one full path to a file per line. Any file in this list would not be pushed to DuraCloud. Directories in the list would be excluded as well, along with all files and subdirectories in each directory. A more complex implementation would allow for using wildcards to indicate files that should not be included. The capabilities of rsync should be reviewed in order to determine the most convenient syntax for the exclusions.	2
477	DURACLOUD-736	Loading throbber disappears before content item is loaded.	Not a huge problem, but is slightly confusing if there is a lot of latency: 1. click on a space and then click a content item. 2. notice that the "loading..." throbber flashes and then disappears before the new content is loaded.	1
478	DURACLOUD-737	SyncTool: Preserve local file timestamp information	Based on feedback from the State of North Carolina, they would like the sync tool to preserve the local timestamp information for individual files (both the creation and modified dates/times). They would like this information passed into DuraCloud via the sync tool, available for viewing within DurAdmin, and then retrieved/restored properly by the retrieval tool when content is downloaded.	3
479	DURACLOUD-738	Java 7 support	This task is to transition DuraCloud to be built and run using Java 7. There are a few reasons for this transition: 1. Oracle has announced that Java 6 will hit end-of-life in Nov 2012 (http://www.oracle.com/technetwork/java/eol-135779.html) 2. Java 7 has been available for over a year, and is becoming more prevalent 3. Java 7 has new capabilities. One specific need is the capability (in NIO) to read/write file creation, modification, and access timestamps 4. We need to support customer use of Java 7 with our client-side tools (e.g. sync tool)	3
480	DURACLOUD-739	Timestamp collection utility	This task is to create a utility which can loop through a set of directories and capture the timestamp information (Creation Date, Modified Date, Access Date) on each of the included files. That timestamp info should be written to a file (so that it can be archived locally) as well as written as metadata to each files's corresponding DuraCloud content item. (The expectation here being that the Sync Tool has already done the work of moving each file to DuraCloud.) The capability to capture time stamp information was added to the NIO package in Java 7 (http://docs.oracle.com/javase/7/docs/api/java/nio/file/attribute/BasicFileAttributes.html), thus this utility will need to be built and run using Java 7. This task is in direct response to needs expressed by the State of North Carolina archives and libraries. Everyone in their offices uses Windows (either 7 or XP), so this utility needs to be tested on Windows.	2
481	DURACLOUD-741	Bit integrity comparison reports have incorrect header	The reports generated by Bit Integrity Checker and Bit Integrity Bulk have incorrect headers. The header value is "space-id <\t> content-id <\t> MD5" but should be something like "space-id <\t> content-id <\t> MD5-file <\t> MD5-provider <\t> comparison-result".	1
482	DURACLOUD-742	UI: Number of items displayed count not always updated properly	The "Showing X of X" text at the top of the Content Item pane (with a space selected) is not always correct. Examples: 1. Select a space, notice text at top of content list: "showing 1 - 200 of X", switch storage provider, notice text is not cleared 2. Select a space, notice the "showing X of X" text. Select several content items and delete them. Notice that the "Showing " numbers don't change.	2
483	DURACLOUD-743	Sync Tool: Clean up log output	The Sync Tool currently logs a variety of "errors" when it is running normally. Many of these log statements have to do with calls to determine if a file currently exists in DuraCloud. The attached file includes errors showing up in the log (provided by users at UTK.) This task is to clean up the logging outputs so that only actual errors are logged as such.	1
484	DURACLOUD-745	DuraStore: Skip sending event messages for content in administrative spaces	When any event occurs in DuraStore (space/content is added/updated/deleted) a message is sent. This task is to update that mechanism to skip sending those messages for the "administrative" spaces in DuraCloud (x-duracloud-admin, x-service-out, x-service-work). There is currently no need for updates on administrative files, and dealing with all of the messages generated by admin file updates wastes effort.	1
485	DURACLOUD-746	Sync Tool: Add option to not overwrite changed files	This task is to add an option to the Sync Tool which ensures that it does not overwrite any existing DuraCloud content. This means that when a file on the local file system, which has previously been added to DuraCloud, changes, it is not simply uploaded again. There are a few choices that need to be made for this new feature to be added: 1. When a file already in DuraCloud changes on the file system, is the updated file transferred to DuraCloud or not. If so, the original file will likely need to be renamed to prevent collision. 2. How is the list of updated files communicated to the user. Are they printed to the console (stdout), or written to a file? Is this consistent between interactive mode and run-to-completion mode? 3. How does this feature impact the upload tool? A comment from the State of North Carolina about how they envision this feature: "Our ideal upload tool would allow us to specify a local directory to update in the cloud, and the tool would then identify and upload new files (new files defined by file name and path), and alert us to any files that have changed (same file name and path but new checksum hash) and not upload the files that have changed (or upload them in a way that doesn't overwrite the files in the cloud). We would want to make sure that the process of uploading content is kept totally separate from any risk of editing existing content in storage."	2
486	DURACLOUD-747	Dup-on-Change: Add an option for default duplication setting, to handle newly created spaces	With the updates to the Duplicate-on-Change service for 2.1, it no longer duplicates newly created spaces. This is fine for multi-tenant instances, but not for enterprise instances, where users may be adding new spaces frequently.	2
487	DURACLOUD-748	Download Multiple Items as a zip or tar archival file.	This feature sprung into my mind: it is not driven by user requests, but it seemed that it would be very handy especially once we have a search index for listing content items. The idea is that it might be useful to be able to download multiple content items in one zip archive. This would be especially useful if one needed to extract a large number of small files. You could select items either based on a search query or a list of contentIds. I was envisioning it would work both at the durastore as well as duradmin levels. You would issue a request for an archive and immediately receive a handle. One could optionally specify an email address. The handle would enable you to get the status of the request. When the request is complete, if an email address had been specified, the user would receive an email with a link to the download page. Otherwise the download URI would be available via a status query using the request handle. Once the archive had been downloaded and verified the user could then make a call to remove download site. If no call was made after a standard expiration time, the download would be removed automatically. Regarding limits: Initially the total size limit would be constrained by the available server side disk space. Obviously this is not going to work for extremely large downloads, ie greater than 100 GBs.	5
488	DURACLOUD-751	UI: Large bit integrity reports cannot be displayed	For spaces with a large amount of content, the UI option to view the service report in a nice table does not work. This option is presented in the UI on the green health check bar, as well as on the services tab. If the "report" link is selected for one of these large spaces, the UI brings up the white display box, then the user waits for a rather long time (while DurAdmin is attempting to read the entire report file into memory so that it can be displayed) and eventually an error is shown indicating that not enough memory is available. To fix this, the size of the report file should be checked. If the size is over some reasonable threshold, the report link should perform a file download (of the raw report) rather than attempting to display the chart.	1
489	DURACLOUD-752	UTF-8 Support in Content Properties	It appears that UTF-8 characters are not supported in Content Properties. Test: add the following properties to a content item: ĽĔĘŔĴêť ﯔﷲﮘﺉﺊﺴﻅ ᾡὉᾧ―♫♪♫	1
490	DURACLOUD-754	AWS Glacier	This feature is to expose AWS Glacier as an option for content storage. http://aws.amazon.com/glacier/	6
491	DURACLOUD-755	Secondary storage provider should be read only via the UI	As the synchronization of content between storage providers becomes automated (DURACLOUD-642), it becomes unnecessary for users to edit content in their secondary provider. In order to maintain an assurance that content remains consistent between providers, the ability for users to make changes at their secondary storage provider should be removed. They would still be able to view and download all content, but no add/update/delete operations would be allowed. This task is to update the DurAdmin UI to ensure that users with role user/admin/owner do not have write access to secondary stores.	2
492	DURACLOUD-756	Message consumers disconnecting	This issue is that for a few different reasons the JMS connections which are used to listen to messages from the brokers hosted by DuraStore/DuraService can disconnect and cease to receive any messages. This has been recorded in the Duplicate on Change service, as seen in the stack traces in the attached file. This results in the Duplicate on Change service being rendered essentially useless. This is likely also to be the cause of similar issues where the Auditor stops recording changes in the audit log.	1
493	DURACLOUD-757	Content properties are being overwritten when content stream changes.	When the content of an item changes locally and is resynced to duracloud, the custom metadata (properties and tags) are wiped out. To reproduce follow these steps: 1. add a file using the upload applet 2. add a custom property and tag to the content item. 3. change the file on your local machine by adding text to the end of it. 4. resync it using the upload applet. 5. verify that the content metadata is no longer there. Another symptom of this problem: if you log in as userX on the initial content item creation and then login as userY when uploading the changed content item, notice "creator" property changes from userX to userY.	1
494	DURACLOUD-758	UI: Check box near space name	In DurAdmin, each space name has to its left a checkbox. Users expect that selecting the check box will perform the same action as selecting the space name text. Instead, selecting the checkbox (of one space or more) brings up a multi-select display. A user from UCAR/NCAR was recently caught by this issue, and could not understand why the listing of content items was not showing up until they contacted Carissa for assistance. This task is to discover and implement an alternative to the current UI functionality described above which will be more easily understandable. From Carissa: Three users within the past month have had the misunderstanding that selecting the checkbox next to a space name would then allow them to view their content. Each time I had to field either a support email or call and explain that they should *not* select the checkbox next to the space name to view the content, add content, edit content, etc. but that they should only click on the name to select/highlight the space to view the content. Ideally either clicking directly on the checkbox or the space name will select the space (add the checkmark) and both actions will behave identically: they will display the content of the space with all the other associated actions. The two actions that we lose by doing this are the "delete selected space" and "edit properties" on the space level. I believe we should determine another way for those options to appear.	2
495	DURACLOUD-759	Sync Tool: Hidden password option	This task is to provide an option for the Sync Tool which will allow users to supply their DuraCloud password in a way other than via the command line options. The reason this is an issue, is because the full command used to run the sync tool remains available in the process list of the machine on which the sync tool is running. The suggested fix for this concern is to allow users to define an environment variable which contains their DuraCloud password, which would be picked up by the Sync Tool if the -p option is not provided.	1
496	DURACLOUD-760	Make use of durable subscriptions for messaging connections	This task is to introduce the use of durable subscriptions in place of standard topic subscriptions in order to have a better assurance that messages are not lost in the event of an interruption of a JMS connection. The failover directive, which is currently in use, allows connections to be re-established, but there is the potential for messages which were sent during the time when the connection was down to be missed. More information can be found here: http://activemq.apache.org/how-do-durable-queues-and-topics-work.html http://activemq.apache.org/manage-durable-subscribers.html	2
497	DURACLOUD-761	Auditor not receiving messages	The auditor, as well as the service reporter (both part of DuraBoss), are not receiving messages in 2.1.1 instances.	2
498	DURACLOUD-763	Notification email address validation needs work	As reported by Dan Galewsky, the regex being used to validate the From email address in DuraBoss (which is used to send notification emails on events) does not properly handle all valid email addresses. In particular, the example provided is that setting this value in init.properties: duraboss.notification.0.originator=user@austin.utexas.edu causes this error fromAddress not valid notification: user@austin.utexas.edu The issue is that the regular expression used in AmazonNotificationFactory.java fails on some valid email addresses, like the one above.	1
499	DURACLOUD-764	Ability to easily share links to DuraCloud content	There should be a simple way to share content that is stored in DuraCloud. Something like a "share" button/feature for spaces/content items that are stored in DuraCloud would likely work. Initially, this could just be for content in open spaces. This would keep users from having to manually find the storage URL for a content item and then find a way to share it. Instead, the user could be presented with a pop-up window that could be configured to send an email or, perhaps, generate some embeddable html code. This feature would make DuraCloud a convenient platform for sharing large files that were otherwise difficult to share, as they are too large to email or pass around via IM. This feature would be notionally very similar to Dropbox Links (https://www.dropbox.com/help/167/en).	2
500	DURACLOUD-766	SyncTool error	This issue was reported by Bryan Beecher on Dec 20, 2012. He was, at the time, using the latest Sync Tool from trunk: I'm seeing a seemingly unending series of errors like this: at java.util.SubList.add(AbstractList.java:652) written to stdout/stderr in the window where I run synctool. I don't see anything obvious in any of the synctool log files to account for the error, and I can't read the status display from the synctool to tell if things are working despite the error, or if the train has gone off the tracks. The errors persist across restarts. I've tried it a couple of times now, including removing the contents of the log directory. Reverting back to version 2.0.0 cleared up the problem.	2
501	DURACLOUD-767	Use S3 bucket tags for system space properties and remove feature allowing users to set space properties.	The new Glacier storage provider brought back to light the missing ability in Amazon S3 to set properties at the bucket level. Up to this point, we have used a file in each bucket to store DuraCloud space properties, but this will not work in the Glacier model, as that file will be archived to Glacier along with all content. As no current customers are making use of space properties, this issue can be resolved in two steps: 1. Make use of S3 bucket tags to store the properties generated by DuraStore for each space created 2. Remove the SetSpaceProperty feature Step 2 is necessary because of the limitations of S3 bucket tags. Only 10 tags are allowed, none more than 256 characters. While it would be possible to allow for a very limited number of metadata/tags provided by the user using a limited character set, the fact that this feature is not currently in use suggests that it is simply not needed.	6
502	DURACLOUD-768	Bit Integrity Tools: Service report not created for empty spaces	When running the Bit Integrity Tools service over a space with 0 content items, it completes successfully with a link to a report file, but that report file does not actually exist. Note that this is the same problem which was reported and resolved in DURACLOUD-704. It appears to have been re-introduced. This task is to ensure that bit integrity reports get generated and stored in x-service-out for empty spaces.	1
503	DURACLOUD-769	Remove Space Properties and Tags in Duradmin	In order to support Bill's strategy for glacier, see DURACLOUD-767	1
504	DURACLOUD-771	UI: Expose the ability to retrieve content items from Glacier in DurAdmin	DURACLOUD-754 adds the ability to use Glacier as a storage provider. Glacier requires additional steps to be able to retrieve content (as compared to other DuraCloud storage providers). This task is to expose those additional steps via the UI.	2
505	DURACLOUD-772	Glacier: provide a way for users to perform limited file retrievals	The Glacier storage provider allows for very low cost storage, but retrieval costs can add up very quickly. This task is to limit the ability of users to request content retrievals in such a way as to minimize the risk of high costs.	2
506	DURACLOUD-773	Allow Glacier to be used as a primary storage provider	This task it to allow Amazon Glacier to be the primary provider in DuraCloud, rather than Amazon S3.	3
507	DURACLOUD-774	Pull the UpSync tool into the DuraCloud baseline	This task is to copy the UpSync tool from the DfR codebase into the DuraCloud codebase in order to expose it as a DuraCloud feature and in order for ongoing development to occur. Once transferred, the tool should be tested with DuraCloud by both technical and non-technical staff to ensure that it can be easily installed, it functions properly, and that there are no usability issues.	4
508	DURACLOUD-775	UI: Make client-side tools easily available	This task is to update DurAdmin to provide very obvious links which allow users to download the client-side tools (Sync Tool, Upload Tool, UpSync, Retreival Tool).	2
509	DURACLOUD-776	Option for Retrieval Tool to set time stamps on retrieved files	DURACLOUD-737 allows the Sync Tool to preserve local time stamp information in DuraCloud content item properties. This task is to add an option to the Retrieval Tool which will set the time stamps of files that have been retrieved (assuming that those files have time stamp information recorded in DuraCloud).	2
510	DURACLOUD-778	UI: Handle Glacier storage state errors gracefully	When content is stored in Glacier, attempts to download, view, copy, or edit the properties of that content will result in an error (http status 409). This task is to provide users with useful feedback when they attempt any of these actions on a content item in Glacier. More specifically, when a call is made on a Glacier content item via the storeclient, a ContentStateException is thrown. (If the content item was recently added or has been retrieved from Glacier, it will simply download as expected) This task is to show the user an appropriate error message indicating that the content is in Glacier. For now, that error message can indicate that the user should contact DuraCloud support if they wish to retrieve that content item.	1
511	DURACLOUD-780	sync tool include top level directory option/flag	After requests from several trial and customer users, the synchronization tool should provide a configuration option/flag in which the entire directory path is included upon upload. Currently, the top level directory is ignored and not persisted in the naming of the content items that are uploaded via the sync tool. This new flag for the sync tool would enable the entire folder structure to be included in the name of the uploaded content.	1
512	DURACLOUD-781	SyncToolUI advance config	This task is to add advanced config options to the sync tool UI. In particular, the user should be able to set the connection port as well as have the ability to select a "Sync Deletes" option.	2
513	DURACLOUD-784	Update Duracloud to the latest ActiveMQ	DuraCloud has a dependency in the POM to ActiveMQ 5.2, an old version. The version does not formally support Java 7, supported in DuraCloud 2.3. In particular, specialized settings are needed to work with Java security. DuraCloud 2.3 has been tested using ActiveMQ 5.7 and 5.8 so the update should require little effort and have little impact.	1
514	DURACLOUD-786	SyncToolUI: Allow for removing sync directories in initial config	Currently, when walking through the initial setup wizard of the SyncTool UI, if you select a sync directory or file accidentally, there is no way to remove it except to cancel and start the wizard over. This task is to allow selected directories or files to be removed, as they can be in the config update area.	2
515	DURACLOUD-787	Inconsistent Glacier Errors for Root Users in Duradmin	Root users attempting to edit, copy, or edit metadata on a file in Glacier get inconsistent failure errors.	1
516	DURACLOUD-788	SyncToolUI: Notify user that restart is required after change to configuration.	Configuration tab: When changing the configuration, this does not take effect until there is a stop/start cycle of the tool. This should be indicated to the user, and an option should be provided to perform a stop/start now or to wait.	2
517	DURACLOUD-790	UI on IE issues	The following UI issues were discovered while using DurAdmin via IE during testing for the 2.3 release: 1. Attempting to retrieve a Glacier stored content item in IE does not show the expected notice message (that is seen in Firefox and Chrome), but instead displays the message "This website is too busy to show the webpage." 2. Filtering on content item names does not work in IE 3. In IE, when a space refresh is performed, the selection of the space is forgotten, and the display responds as if the Spaces tab were selected.	2
518	DURACLOUD-794	SyncToolUI: Allow clearing of errors	This task is to provide a way, in the SyncToolUI, to clear any errors that have been noted on the errors list.	1
519	DURACLOUD-796	SyncToolUI should expose pause/resume functions of the synctool.	NULL	2
520	DURACLOUD-798	Synctool UI should stop promptly when the user clicks "stop"	Currently the tool remains in the "stopping" state until the currently active transfers are complete.	2
521	DURACLOUD-800	SyncToolUI: Better feedback for user without write permissions on space	The SyncToolUI should be updated to provide better feedback to users when they have selected a space to sync to which they do not have permissions to write to. Since there is not a simple way to collect this information up front, this task is to pay attention to exceptions that are coming back from a sync activity to determine if it is likely that they are a result of lacking permissions, then provide information to the user that will allow them to be aware of he problem and a suggested resolution.	1
522	DURACLOUD-801	Shibboleth: Add 403 Access Denied Page	This improvement is dependent on the Shibboleth branch. A DuraCloud-branded 403 page should be rendered when an authenticated user who is not authorized to access DuraCloud tries to log in.	1
523	DURACLOUD-802	Shibboleth: Logout	This improvement is dependent on the Shibboleth branch. When a user logs out, he/she should be redirected to a page that specifies that complete logout requires closing the browser (ala the MC).	1
524	DURACLOUD-803	DuraCloud instance hitting 100% disk utilization	Initial issue: - After running synctool on a set of 200,000 files, the sync rate slowed from about 3,000 files transferred per 10 minutes to about 300 or less per 10 minutes. Reason for the slow down: - The reason the rate dropped, was because the instance storage space had become 100% consumed. The major culprit was the catalina.out and catalina.out.1 files in /mnt/log/tomcat6 Reason there isn't much space available: - As it turns out, this EBS instance, as well as all other EBS instances, no longer include ephemeral storage space by default. It was expected that /mnt was mounted to a large ephemeral storage drive, but that is not the case. - Further info: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#using-instance-storage	1
525	DURACLOUD-804	Use JClouds library for all calls to OpenStack providers	The current OpenStack storage provider implementation uses both the client originally provided by Rackspace (which has now been discontinued) as well as the JClouds library (which is now the client recommended by Rackspace for Java development: http://docs.rackspace.com/sdks/guide/content/java.html). This task is to update the OpenStack and Rackspace storage providers to use only JClouds.	5
526	DURACLOUD-805	Can only perform a bulk action (e.g. deletion) on a maximum of 2 spaces at a time	Today I went in to cleanup some old demo/test content in 'dspacetest.duracloud.org'. When I selected multiple spaces, I found that no matter how many spaces I selected, the total always read "2 space(s) selected". Even worse, I selected a total of 5 spaces and clicked "Delete Selected Spaces" button. Only the first 2 spaces I selected were deleted. The other 3 were kept. It seems as though the multi-select option only captures the first two spaces you select...after that, any other selections are ignored. Carissa has verified that this also seems to occur on demo.duracloud.org.	1
527	DURACLOUD-806	Recreate storage provider connections	For some reason, on occasion, all calls to an SDSC provider will start to return 401 responses. When this happens, no calls go through to SDSC. This issue can currently be resolved immediately by reinitializing the instance. The issue also appears to resolve itself after some amount of time has passed. Since it appears that recreating the SDSC connection (which occurs on reinit) resolves the issue, this task is to update DuraStore to be able to recreate those connections. Ideally this would happen when we start to see 401 failures, but another option would be to simply recreate the connections on a set schedule.	3
528	DURACLOUD-808	Space size alerts	It would be useful for users to be notified when their space reaches a configurable size. This feature would allow the users to configure this information via duradmin.	3
529	DURACLOUD-809	When starting sync in SyncTool UI throws an error, the error message is ugly.	A more compact error description should be displayed - the large amount of text coming back in the message is breaking the look and feel.	1
530	DURACLOUD-810	Update GlacierStorageProvider to retrieve content item storage class	This task is to update the calls for getting content properties in the GlacierStorageProvider to include a call which will retrieve the storage class of each object, and include that information in the content item details. This will make it possible for users to know if a content item is stored in Glacier, or if it has been retrieved back to S3. Information about the call that needs to be made to retrieve the storage class can be found here: https://forums.aws.amazon.com/message.jspa?messageID=312132	1
531	DURACLOUD-811	Print URL to set up screen on the command line when the Sync Tool is executed in GUI mode	Especially on Linux, the automatic display of the Sync Tool setup/admin screen is not reliably displayed in a browser. Also, when is does display if there are many browser windows open it is hard to find. It would be helpful to display the URL as a message when the Sync Tools is started from the command line. It would also be helpful to display the URL as part the Tray icon in some fashion.	1
532	DURACLOUD-813	UI: Remove upload tool applet, reinstate single file upload	Now that there is a solid UI as part of the Sync Tool, and because of the seemingly constant issues with java applet technology, the multi-file upload applet should be removed from DurAdmin and the single file upload option should be re-instated. This work should occur alongside DURACLOUD-775, which provides easy access to the Sync Tool.	3
533	DURACLOUD-823	Sync Tool: Provide feedback for Java issues	There have been a variety of customer questions and issues reported in using the Sync Tool which mostly boil down to the user's system not having the necessary Java version or not having Java installed at all. This task is to provide actionable feedback to users as they attempt to use the Sync Tool, which will allow them to more easily resolve the problems and get the Sync Tool operational.	4
534	DURACLOUD-824	Sync Tool: Ability to restart on system reboot	This task is to provide a way, on Windows, Mac, and Linux systems for a Sync Tool that is running on shutdown to restart and continue running after a system restart.	3
535	DURACLOUD-825	S3StorageProvider: Remove dependency on a particular access key	The S3StorageProvider makes use of the AWS access key as a prefix for bucket names, just to guarantee that the names are unique. The listing of buckets is also limited to only those with a matching access key. Now that AWS does not make the secret key (essentially the password) available, if we were to lose the secret key, we would have to create another credential pair. If we were to do that, though, the S3StorageProvider would show an empty space listing because the new access key will not match the bucket prefix. This also currently makes it impossible to use the AWS IAM service, which is a best practice. This task is to allow the S3StorageProvider to display and work with buckets regardless of the access key prefix. The prefix should still be used, since uniqueness is still necessary, and should still be stripped before displaying the space names for users. However, rather than requiring a specific access key, any key should work. Since this match will likely require a regex: According to AWS docs, the Access Key ID is a 20-character, alphanumeric sequence.	2
536	DURACLOUD-826	Retrieval Tool: Use logback for logging	In DURACLOUD-774 (in particular, the changes in code r1072) changed over the Sync Tool from using log4j to using logback as the underlying logger. This task is to update the Retrieval Tool in the same way.	1
537	DURACLOUD-827	add sync tool prepend source path (include top level directory) flag to synctoolui	NULL	1
538	DURACLOUD-828	Retrieval Tool: Hidden password option	DURACLOUD-759 added a hidden password for the Sync Tool. This task is to make use of the same environment variable to pick up the password for the Retrieval Tool. This will help to keep the functionality of the two tools in sync.	1
539	DURACLOUD-831	Support multiple instances of synctool ui on one machine	Currently the synctoolui assumes only one user per machine will be using the synctool. If this assumption is incorrect, we may need to look into adding port negotiation into the installer mix to enable multiple installs of the synctoolui on a single machine.	2
540	DURACLOUD-832	RetrievalTool NullPointerException when using all spaces option (-a)	The retrievaltool throws a NullPointerException when using the "all-spaces" or "-a" command line option and the "spaces" or "-s" option is not specified, which the "spaces" option shouldn't need to be when using "all-spaces". The culprit is in RetrievalTool#startRetrievalManager at the line: boolean createSpaceDir = retConfig.getSpaces().size() > 1; because RetrievalToolConfig#getSpaces returns null when the "spaces" command line option is not used.	1
541	DURACLOUD-833	synctoolui: unchecking options in configuration tab causes 404 error	The problem: the controller dependends on the presence of property name in query param while uncheck form fields are not sent as empty values.	1
542	DURACLOUD-837	List MD5 option for space listing function of Retrieval Tool	It would be nice to be able to optionally list all the md5s as well as the raw list of contentIds for a space.	1
543	DURACLOUD-838	List contents of space from a "pre-chunked" point of view (Retrieval Tool)	It would be handy to be able to optionally list a space contents as it will appear when retrieved. That is, only list contentIds as they will appear after they are restitched (do not show chunks or manifests). It's a kind of dry run for the retrieval tool in a way.	1
544	DURACLOUD-839	Retrieval Tool: Verify chunked manifests when generating prechunked listing	It would be nice if we could tell the retrieval tool to verify that the manifest points to a complete set of chunks and that the chunk md5s match the manifest md5s. This will be a handy addition to https://jira.duraspace.org/browse/DURACLOUD-837 and https://jira.duraspace.org/browse/DURACLOUD-838 In verify mode, it would probably be a good idea to add a couple of columns to the output: verification status (success, failure) and an explanation column where there is a problem.	1
545	DURACLOUD-841	Image gallery viewer	In a cloud service strategy meeting on Aug 6, 2013 (https://wiki.duraspace.org/display/DSPINT/2013-08-06+-+Cloud+Service+Strategy) there was a discussion about a need expressed by UNC to have a way to view images in DuraCloud as a gallery, in order to aid in the finding of images in their collection. I was thinking that this could be accomplished with an html/css/javascript app that is stored in one of the user's spaces, and allows them to select a space (if that's actually necessary, could be hard coded to the one that is their image library) and view a set of images. The javascript would need to make calls to the DuraCloud REST API to get the list of content items and display each one. The images would likely be scaled down a bit, to make browsing easier, and clicking on them would display the (slightly) larger size. The REST API would also be used to allow paging through the content items (and not trying to display too many at once). The option to specify a prefix for filtering would need to be there, and the user would need to be able to see the full content ID of the item they are viewing, since that will be the link back to the full size archival image on their internal system. There is an expectation that the UNC folks will be generating the necessary low-res images that would be viewed through this gallery viewer, and would load those images into a DuraCloud space.	2
546	DURACLOUD-842	DurAdmin: Uploaded files should include same metadata as generated by Sync Tool	When adding files, the Sync Tool captures metadata for the files. DurAdmin should capture the same metadata. The metadata items created by the Sync Tool includes: - creator - content-file-last-accessed - content-file-path - content-file-modified - content-file-created	1
547	DURACLOUD-843	Make use of S3 Server Side Encryption	This task is to update the S3StorageProvider to make use of the Server Side Encryption feature provided by S3 (http://aws.amazon.com/about-aws/whats-new/2011/10/04/amazon-s3-announces-server-side-encryption-support/) Implementation details: http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html Storing content with encryption should be the default. Another task will be needed to update existing content in S3 to make use of encryption. This will likely need to be done by performing a copy action on all existing content such that the copied item is encrypted as it is written to S3.	2
548	DURACLOUD-844	Sync Tool: Uninstalling with app running (on Windows) pops up error dialog	If the sync tool is uninstalled while the application is running, the uninstall works properly, the application is stopped and all files are removed as expected, but an error dialog is popped up stating "Program ended with an error exit code". This popup does not show up if the application is not running when the uninstaller is run. The dialog is popped up when the command: "taskkill /f /t /im duracloudsync.exe" is run by the uninstaller. In testing for the 2.4 release I found that if the taskkill command is run without the "/f" force flag, the error dialog does not pop up and the exe process stops (the icon is removed from the task bar), but the java process continues to run, so the application is still available. If the command above, with the force flag is then run, the java app is stopped without a popup dialog. This is why a second call was added to the uninstaller, to try running without the force flag first. Unfortunately, this did not work, the error dialog remains.	2
549	DURACLOUD-846	Update to the latest JClouds	In DuraCloud release 2.4 an issue with the JClouds library that was used (the latest at the time) was that it would not perform reauthentication when the auth token expired. This required a work around in DuraStore. The issue has now been fixed: https://issues.apache.org/jira/browse/JCLOUDS-178. This task is to update to the latest JClouds library when possible.	1
550	DURACLOUD-847	SyncTool/StoreClient failing when attempting to connect to a DuraCloud instance with a configured Glacier provider	From ucar customer, when attempting to run the sync tool (version 2.1.0) against a DuraCloud 2.4 instance with a Glacier storage provider configured, the following error appears: Starting up the Sync Tool ...Exception in thread "main" java.lang.RuntimeException: Could not create connection to DuraStore due to Error retrieving content stores. null at org.duracloud.sync.util.StoreClientUtil.createContentStore(StoreClientUtil.java:40) at org.duracloud.sync.SyncTool.startSyncManager(SyncTool.java:148) at org.duracloud.sync.SyncTool.runSyncTool(SyncTool.java:292) at org.duracloud.sync.SyncTool.main(SyncTool.java:355) Caused by: org.duracloud.error.ContentStoreException: Error retrieving content stores. null at org.duracloud.client.ContentStoreManagerImpl.getStorageAccounts(ContentStoreManagerImpl.java:164) at org.duracloud.client.ContentStoreManagerImpl.getPrimaryContentStore(ContentStoreManagerImpl.java:118) at org.duracloud.sync.util.StoreClientUtil.createContentStore(StoreClientUtil.java:37) ... 3 more Caused by: java.lang.NullPointerException at org.duracloud.storage.domain.StorageAccountManager.initialize(StorageAccountManager.java:50) at org.duracloud.client.ContentStoreManagerImpl.getStorageAccounts(ContentStoreManagerImpl.java:152) ... 5 more	1
551	DURACLOUD-848	Bulk Select/Delete Tool	Several customers have requested the need for a tool that will allow them to perform a bulk delete of a subset of content in a space. They need a tool that will allow them to, first, specify which content to delete from a space and then, second, perform a bulk delete for that content.	1
552	DURACLOUD-851	Add CORS headers to apache config in new AMI	Add the CORS headers to the apache config when the new ami is built: See https://jira.duraspace.org/browse/DPS-23 for modifications. This change enables api calls to be made from javascript hosted on other domains.	1
553	DURACLOUD-857	upgrade for spring 4.0.2	NULL	3
554	DURACLOUD-858	Upgrade to tomcat7	NULL	2
555	DURACLOUD-859	Modify SpaceWriteAccessVoter.java so that the primary storage provider is inferred when no storeId is specified in the URL.	NULL	1
556	DURACLOUD-860	Remove all services dependencies from duradmin	NULL	2
557	DURACLOUD-861	Modify duradmin to control streaming service via ContentStore interface and space properties	NULL	2
558	DURACLOUD-862	Remove services system	This work is to remove all of the services from DuraCloud. This should include removing DuraService, the entire set of existing services, and components (like the executor) which support or depend on services.	3
559	DURACLOUD-863	Remove ActiveMQ	This work is to remove the use of ActiveMQ in DuraCloud. Storage events should be now captured by the AuditStorageProvider.	2
560	DURACLOUD-864	Update media streaming to work without Media Streamer service	The Media Streamer service is being removed. Media streaming still needs to work. This task is to allow the existing S3StorageProvider streaming Tasks handle the work the service used to manage.	2
561	DURACLOUD-865	Create AuditStorageProvider to push AuditTasks to audit queue	An AuditStorageProvider needs to be created which is a decorator for StorageProvider. This new class will capture read and write events and write them to an SQS queue as AuditTasks, to be processed later. Note that this item overlaps with DURACHRON-35.	5
562	DURACLOUD-866	Set up local logging to capture read and write storage actions	The AuditStorageProvider already captures read and write events. This work is to have that information written out to local log files	1
563	DURACLOUD-867	Space listing not being displayed in Chrome browser	When loading the DurAdmin application in the Google Chrome browser, the list of spaces is not displayed. If there are multiple providers configured, selecting a different provider from the dropdown, and then returning to the primary provider allows the spaces listing to show up. If there is only one configured storage provider, there is no clear way to get the spaces listing to display.	1
564	DURACLOUD-868	Upgrade from Maven2 to Maven3	Upgrade from Maven2 to Maven3. Update POMs to be compatible with Maven3. Fix all build warnings and errors.	3
565	DURACLOUD-869	Sync Tool: add content ID prefix option	In a discussion with NCDCR, they noted that they would like to be able to break up their data set, and perform syncs at lower levels in their path hierarchy, but are prevented from doing so because they would like the full directory path to be included in the content ID. Currently, the only way to do this is (in a Windows environment) to either run the Sync/Upload tool from a high level directory, or move content around. Moving content around, in their production storage system is not practical. The new feature requested to resolve this problem: Allow for a prefix option, that would be prepended on all content items being transferred. To keep this simple, the prefix provided would apply to all files in all specified content directories.	2
566	DURACLOUD-873	SyncToolUI: Option to set threads	The UI for the SyncTool should provide an option to set the number of threads used for processing. This can just be a drop-down box on the configuration screen.	3
567	DURACLOUD-874	SyncToolUI: Option to include prefix	DURACLOUD-869 added a prefix option to the SyncTool. This task is to provide a way for users of the UI to make use of that new feature.	1
568	DURACLOUD-876	Create 3.0.1 AMI	Updates: - Use 3.0.1 code baseline - Add lumberjack - Update ssh users - Root pw - SSL cert update	1
569	DURACLOUD-877	Provide a way by which users can determine the optimal SyncTool thread setting	Currently, to determine the ideal number of threads to maximize throughput, users of the SyncTool need to run the SyncRate tool with a variety of settings, record the speed values, then compare them. This is time consuming and difficult. This task is to make determining the thread number for optimal throughput as simple as possible.	3
570	DURACLOUD-879	Update mimetype determination method	The MimetypeUtil class in common currently uses FileNameMap (http://docs.oracle.com/javase/7/docs/api/java/net/FileNameMap.html) to determine the mimetype of a file based on its file extension. This class is used by the SyncTool to generate mimetype values for files being transferred to DuraCloud. The current method depends on a mapping file found in each individual user's JRE, make it difficult to update. A better option would be to use MimetypesFileTypeMap (http://docs.oracle.com/javase/7/docs/api/javax/activation/MimetypesFileTypeMap.html) for this purpose, which would allow us to include our own mapping file (and therefore be much more comprehensive).	1
571	DURACLOUD-880	Media streaming URL not being displayed in space properties	When media streaming is turned on for a space, a property is added to the space tags called "streaming-host" which provides the host value needed to stream the files in the space (this change was made for DURACLOUD-864). Unfortunately, as of the 3.1.0 release, DurAdmin is not displaying the streaming host value as one of the space properties, so users cannot actually use streaming even after it has been turned on.	1
572	DURACLOUD-881	Cannot configure or turn off logging of Retrier in ContentStoreImpl	The ContentStoreImpl currently has hardcoded settings for how it logs errors encountered during "retries". Unfortunately, this is not ideal for all use cases of the API. For example, in the Replication Task Suite, we use ContentStoreImpl.getContentProperties() method as a way to tell whether content already exists in DuraCloud. If that content does not exist, we expect to receive a NotFoundException. This lets us perform logic like: "If content exists in DuraCloud, see what checksum DuraCloud reports. If DuraCloud checksum differs from local checksum, re-upload the content." Unfortunately, however, getContentProperties() logs this same warning message to log4j 4 times in a row (1 + 3 retries) if the content does NOT yet exist in DuraCloud: 2014-07-14 16:46:28,336 WARN org.duracloud.client.ContentStoreImpl @ Error attempting to get properties 'ITEM@10673-10.zip' in 'dspace-dev-backup' due to: Response code was 404, expected value was 200. Response body value: null This causes the logs for the Replication Task Suite to actually fill up with unnecessary warnings from the DuraCloud v3.1.1 API. For example, anytime we want to upload NEW content to DuraCloud, we first check if it already exists (which logs that error x 4) before the new content is uploaded. The logged warnings would cause unnecessary worry for anyone looking at the logs -- in reality they are harmless in this scenario, as the content obviously doesn't exist before it is first uploaded. It looks like the WARN message is generated by "doGetContentProperties" here: https://github.com/duracloud/duracloud/blob/master/storeclient/src/main/java/org/duracloud/client/ContentStoreImpl.java#L904 It is then repeated 4 times as a "log.warn" in the default execute() method (which is configured to do 3 retries): https://github.com/duracloud/duracloud/blob/master/storeclient/src/main/java/org/duracloud/client/ContentStoreImpl.java#L216 Ideally, there should be some way to configure the logging in the ContentStoreImpl. I like that this same task is retried 3 times. But in many scenarios, I'd rather it just throw me a final Error if all retries failed, rather than logging the same error message multiple times in a row. If logging cannot be easily configurable, then switching the default execute() method to just do a "log.debug" would be better. That way I can see the underlying error if I turn on debugging, but by default my logs don't get filled with the same errors repeated. NOTE: This was discovered while attempting to upgrade the DSpace Replication Task Suite to use DuraCloud API version 3.1.1 (JARS: storeclient, storeprovider, common). Unfortunately, until logging is cleaned up in some way, it's unlikely I'll release the upgraded Replication Task Suite code.	2
573	DURACLOUD-882	Fixity check green bar not being displayed	The green bar that show the date of the last fixity check is not being displayed in DuraCloud 3.1. This feature needs to be re-instated as the new bit integrity system is put in place to allow users to see the dates of health checks.	3
574	DURACLOUD-884	Better handling for content files added with an incorrect checksum	Currently when a content item is added to a storage provider via a PUT content REST call, if the checksum included with that content item does not match the checksum computed by the provider, a 500 response is generated, and the file itself remains on the storage provider. This task is to: 1. Ensure that if content fails the checksum check, it is either not added to or is removed from the space. 2. Return a response with an error code and body text that better describes the issue.	3
575	DURACLOUD-885	Limit storage provider details	The current DuraCloud REST API call to determine the available storage providers on an instance provides in its response the set of properties from init that are beyond the standard set of StorageProvider values. In the case of the SnapshotStorageProvider, some of these details should not be shared.	1
576	DURACLOUD-886	Restricted Streaming	- DuraCloud would continue to use CloudFront to stream files, but would add an option to limit access to streamed content. There would be two options when setting a space to be streamed: open streaming (which is how we do streaming now) and restricted streaming.(the new feature). - When using restricted streaming, no files would be directly accessible. Instead, to expose a file, a new DuraCloud REST method would be provided which could be called to retrieve a signed URL that could then be used to stream the file. - The new REST call would require providing the username and password of a DuraCloud user with administrative rights. - The new REST call would require an ending date and time parameter, after which the returned URL is no longer valid. - The new REST call would allow for providing an optional parameter indicating the IP address or range of addresses that can be used to access the content. The consumer of this service (UNC in this case) will be responsible for: - Verifying that a user of their system should have access to the content that is to be streamed - Making the call to DuraCloud (with appropriate DuraCloud permissions) to request and retrieve the signed URL - Inserting the URL into an RTMP player (e.g. Flowplayer, JWPlayer, etc) which will allow the user to view the content	8
577	DURACLOUD-887	User IP limits	- DuraCloud would add a new user attribute which defines an IP address or range (using CIDR notation) - Once this attribute for a user is filled out, all future attempts to log in as that user (both in DuraCloud instances and the Management Console) would require that the user be coming from that IP address or range. - Users would be able to update this attribute in the Management Console. - Note: This feature would need to support multiple IP address/range entries, as there could be 2 or 3 specific IPs where the user needs to be able to log in from.	3
578	DURACLOUD-888	Make Audit and Manifest REST methods consistent with other durastore calls	Namely, make sure that storeID parameter can be null and if so, the primary storage provider is used.	1
579	DURACLOUD-890	Problem running the latest synctool installer works on osx mavericks	Michele was having trouble launching the installer 3.1.1 on Mavericks. At first it seemed that the problem was the lack of java. Danny had her install java 8. Still no luck. Danny was able to reproduce: Symptoms: After downloading the latest synctool installer (3.1.1), double clicking on the icon appears to do nothing. The system log is showing this output less than completely helpful output: Sep 29 15:54:30 panna com.apple.launchd.peruser.501[241] (com.bitrock.appinstaller.81696[42342]): Job failed to exec(3) for weird reason: 13 Sep 29 15:54:30 panna Finder[259]: 8837325: Attempting to SIGCONT to pid #42342 failed, with errno=#3, or the process failed to actually start Sep 29 15:54:30 --- last message repeated 2 times ---	1
580	DURACLOUD-894	Make org.duracloud.common.DateUtil threadsafe	Neither parse nor format operations are threadsafe.	1
581	DURACLOUD-895	Synctool fails when the max file size is greater than max java integer value	Using the synctool, uploads of files greater than Integer.MAX_VALUE fail when the max file size is set appropriately high (ie > 2 GBs).	1
582	DURACLOUD-896	Add SyncTool feature which allows for skipping existance checks for new content sets	This feature allows initial transfers of data sets to proceed without the usual checks to see if the content is already in DuraCloud (since we know that it is not.) Instead, all files will simply be transferred. The need for this feature comes from GSU. With a large set of small files (4.7 million under 1 MB), the overhead of doing the checks for each file was slowing down the overall transfer considerably.	2
583	DURACLOUD-897	Add REST API for Accessing Bit Integrity Reports in DuraStore	The rest api should support a call to get the last bit integrity report for a space. It should include, along with the body of the latest report, the status of the report (success if all went well, failure if there are errors) as well as the date the report was completed. The properties should be accessible through a HEAD call, while the content (as well as properties) will be accessed with a GET.	2
584	DURACLOUD-898	Spaces no longer loading in Firefox 33.0+ in duradmin	After upgrading to Firefox 33.0 spaces are not longer loading.	1
585	DURACLOUD-903	Clicking content checkbox unexpectedly deselects after loading the content detail	Steps to reproduce: 1. Load a space in duradmin 2. Make sure no content items are selected. 3. Click on the checkbox next to a content item. 4. Notice that it is initially checked but then unchecks itself while remaining selected.	1
586	DURACLOUD-904	SyncTool: Make sure files restored from change list on re-start are in the current set of content directories	An issue reported by NCDCR: The SyncTool performed a restart and picked up an old change list file even though the set of content directories was completely different. This happened because the SyncTool was started (writing the config) then killed before the old change list could be overwritten. On the next run, the config matched, so a restarted was carried out, pulling in change list files which really should have been ignored. To fix this problem, the proposal is add a check on restart, just after the change list is read in, to remove any files from the change list that don't actually exist in the list of content directories.	1
587	DURACLOUD-910	Clicking on the checkbox next to a space causes the application to hang.	To reproduce the issue: Open duradmin. Select a single space by clicking on the checkbox. Notice that the throbber spins continuously.	1
588	DURACLOUD-911	Failure with space and %20 in content ID on SDSC	Several failures have been noted recently in the mill when attempting to perform a duplication of a content item with a content ID that includes "%20" in the content ID from Amazon to SDSC. (Note that %20 is the url encoding for a blank space.) The attached file is a set of errors from the mill dead letter queue. I was able to reproduce the issue by attempting to create a content item in SDSC with the name: "test-file spaces%20percents.txt". Note that I did not see an issue with "test-file spaces.txt" or "test-file%20percents.txt", so it would appear that the combination of the two characters (an actual space, and "%20") is necessary to provoke the issue.	2
589	DURACLOUD-912	Copy dialog in UI has multiple progress spinners next to space selection	As shown in the attached image, when performing a copy via DurAdmin, multiple spinning circles to indicate loading or progress are shown next to the space drop-down selection. This becomes more of an issue when the storage provider selection is used to change the selected provider. Previously, there was one of these spinners, which was present as the list of spaces was generated and then went away one the list was populated. That was great. Now when I transition between storage providers, a string of these are presented (I've counted at least 9) with overlapping text (making the text impossible to read). Some of the spinners do disappear, but 2 seem to remain even after the space listing is fully populated.	1
590	DURACLOUD-914	Remove extra authentication provider from duradmin security config	<authentication-provider user-service-ref="userDetailsSvc"/>	1
591	DURACLOUD-915	SyncTool UI: Add warning about use of the sync deletes feature	It has been noted several times that the sync deletes feature in the SyncTool can be dangerous. It provides the opportunity for a user to delete their entire corpus of content if it's not used carefully. This task is to add a warning dialog when anyone selects the option to turn on the sync deletes feature which describes the dangers and asks them if they are sure this is what they would like to do.	1
592	DURACLOUD-916	Secure streaming cannot be turned back on in DurAdmin	If a space is set to secure streaming it shows up in DurAdmin as having streaming turned on. If, however, you then turn streaming off via DurAdmin and later attempt to turn it back on again, it fails with an error: "Task enable-streamingis not supported". This works fine with a space set to open streaming.	1
593	DURACLOUD-917	Storage report display values are inconsistent	The storage report charts in the DurAdmin display do not show consistent values between the line chart (historical size) and the pie charts. This inconsistency is because one of the two uses GiB (Gibibytes) and the other uses GB (Gigabytes). A GiB is 1024^3 bytes, while a GB is 1000^3 bytes. This issue should be resolved by updating the storage reports to make use of GB rather than GiB. This will provide greater consistency. Note that this is also an issue reported in DURACHRON-79, where the inconsistency is between the storage reports and the snapshot.	1
594	DURACLOUD-919	Durastore init xml is incorrect in the documentation.	This chunk is not right. <storageProviderAccounts> <storageAcct ownerId='0' isPrimary='true'>...</storageAcct> <storageAudit>...</storageAudit> <millDbConfig>...</millDbConfig> </storageProviderAccounts> millDbConfig and storageAudit should be siblings of storageProviderAccounts. See xml document for correct format.	1
595	DURACLOUD-923	Add byte-range option to DuraStore get content REST API request	Discussions with Ilya Kreymer from webrecorder.io have suggested that there could be an integration with that service to allow playback of WARC files directly from DuraCloud storage, which could be quite beneficial for Archive-It users storing content in DuraCloud. In order to make this happen, the DuraStore API needs to support range requests, allowing for only portions of a file to be retrieved from storage, rather than the entire file. This is supported in S3 and OpenStack Swift, so should be something we can support across all providers.	3
596	DURACLOUD-924	Instance not initialized call should return something other than a 503	Each of the DuraCloud apps currently responds to a /init REST call with either a 200 if the app is initialized and ready for use or a 503 if the app is available but not yet initialized. This task is to update this to change the 503 to another code. The reason is that Apache also returns 503 responses as a DuraCloud instance is starting when it is not yet able to make a connection to Tomcat. This results in the Management Console thinking that an instance is ready to be initialized when it is not yet available. Note that this change will require a companion change in the MC to ensure that the correct HTTP code is expected on init.	1
597	DURACLOUD-928	Allow user-level users to see bit integrity notice and manifest button	Users with USER role should be able to see the bit integrity notice and should be able to see and use the space manifest button. Currently these are only available to users with ADMIN role or above.	1
598	DURACLOUD-929	Multiple space selection does not work	In DuraCloud v3.3.0, when multiple spaces are selected, the UI displays as if no spaces were selected. There is no option to delete the selected spaces or edit properties (as there was in 3.2.2 and previous versions) To reproduce: 1. Select a space. 2. Click on the checkbox of another space. 3. The multiple space selection panel does not show up in the detail area.	1
599	DURACLOUD-930	Turning off space streaming clears space permissions	On a space with streaming enabled, setting streaming to off (via the UI) will cause the bucket property which maintains space permissions to be removed. This effectively deletes all space permissions (making the space and the contents of the space unavailable to anyone beyond administrators). Note that this clearing of permissions is not immediately reflected in the UI, as the permissions set is cached.	1
600	DURACLOUD-935	Audit and Manifest links fail in duradmin when space id has a period in it.	To reproduce 1. Add a space with a period (such as "test.space1") and then add a couple of items to it. 2. Wait a minute. 3. Click on manifest or audit links 4. Page will return blank without downloading a file. Problem may be that spring is parsing everything to the right of the period in order to serve up content based on the apparent file type extension. Problem does not exist in durastore - in duradmin only.	1
601	DURACLOUD-936	Durastore incorrectly reports duplicate space.	To reproduce the issue: 1. create a space called "my-space" 2. create a space called "my.space" 3. the second call should fail and report a duplicate space. Note that performing the above calls in reverse order succeeds. Finally, after performing call in reverse order, notice that deleting the my.space results in both my.space and my-space being deleted.	1
602	DURACLOUD-939	Relax check for existing content ID on streaming url request	In order to successfully stream an mp3 file through Cloudfront, the file must be stored with the ".mp3" extension, but the task call to generate the URL must leave the extension off of the file. Example: If a file is named "myAudioFile.mp3", it should be stored in DuraCloud (S3 provider) with that name. To stream that file, the content ID would be specified as: "myAudioFile", without the extension. Right now there is a check in the task calls to create URLs to verify that the specified content ID actually exists. In the example above, this causes problems because the stored file name and the ID provided for the get URL call differ. This task is to change the task code to skip the content ID check altogether. This will also allow for generating URLs for content items which have not yet been uploaded (but for which the IDs are known). Note that I've only tested this with secure streaming, but expect it will work the same with open streaming.	1
603	DURACLOUD-940	Bit Integrity Reports are not showing up in duradmin for secondary providers.	(from an email from bbranan) I was looking at some bit integrity results this morning and noticed in all of the accounts I looked at (colum, icpsr, and demo), that there are no green bit integrity bars on any of the spaces in any secondary providers. I spot checked a few, and there are successful bit report entries for those spaces in the mill db, they just aren't being shown in the UI. When I select a space on SDSC providers in duradmin, it takes a while for the space to load. It appears to be attempting to get the bit integrity report, but it fails to do so. Looking at the logs, I see consistent errors like this: ERROR 2015/09/11 13:26:40 [ajp-bio-8009-exec-25212] (SpaceController.java:117) [populateBitIntegrityResults()] - failed to populate bit integrity results due to error:Error attempting to get bit integrity report properties 'carissa-folder-test' due to: Response code was 500, expected value was 200. Response body value: null org.duracloud.error.ContentStoreException: Error attempting to get bit integrity report properties 'carissa-folder-test' due to: Response code was 500, expected value was 200. Response body value: null at org.duracloud.client.ContentStoreImpl.getBitIntegrityReportProperties(ContentStoreImpl.java:1214) ~[storeclient-3.3.0.jar:na] at org.duracloud.duradmin.spaces.controller.SpaceController.populateBitIntegrityResults(SpaceController.java:109) [SpaceController.class:na] at org.duracloud.duradmin.spaces.controller.SpaceController.get(SpaceController.java:96) [SpaceController.class:na]	1
604	DURACLOUD-941	Streaming: Allow streaming URL requests for content IDs not in storage	In order to stream mp3 files through Cloudfront, the files themselves must have the ".mp3" file extension, but the streaming URL must be requested without the file extension. Currently, there is a check to make sure that a call to get a streaming URL corresponds to a content ID already in storage. In this case, though, the content ID and the "stream" ID don't quite match. This task is to remove the check to see if a content ID is in place before a streaming URL can be requested. This should solve the problem noted above, and make it possible to pre-generate URLs for content files prior to upload, which may be useful as well.	1
605	DURACLOUD-942	Add support for S3 - Infrequently Accessed storage option	According to this announcement: https://aws.amazon.com/blogs/aws/aws-storage-update-new-lower-cost-s3-storage-option-glacier-price-reduction/?sc_ichannel=em&sc_icountry=global&sc_icampaigntype=launch&sc_icampaign=em_148542750&sc_idetail=em_2049897831&ref_=pe_430550_148542750_7, Amazon is now offering the option of a lower cost S3 tier for infrequently accessed data. This task is to transition all DuraCloud content to this new option. The best way to take advantage of S3 - IA is via bucket lifecycle policies. Because of the additional cost of content deletes in the first 30 days, it would be best to set these policies to transition content from S3 Standard storage to S3 - IA storage after the content has been stored for 30 days. This policy should be created when a new space is created (via the S3StorageProvider) As part of this work, the current S3 task which transitions content to Reduced Redundancy storage should be replaced with a task that adds the bucket policy for S3 - IA to an existing bucket. To complete this transition, a script will need to be written which uses this task to update all current accounts. The storage class init parameter should also be removed (from both app-config, and from the MC). It no longer makes sense to support reduced redundancy storage.	2
606	DURACLOUD-946	Use CloudWatch rather than DuraBoss to generate storage reports	Cloudwatch now provides metrics for S3, providing a daily total of files and total size per bucket in an account. We need to use this rather than running DuraBoss for reporting, at least on S3 and Glacier stores. Ideally we can discontinue the use of DuraBoss completely.	5
607	DURACLOUD-949	Beanstalk: Connect DuraStore to MC database	In order to have all of the information that is needed, DuraStore in beanstalk will need to have a direct connection to the MC database. Much of the code for setting up this connection should be already available from the MC or Mill (share/reuse where possible). Options for getting the database connection information: - As system properties specified in JAVA_OPTS (using beanstalk environment variables, note that there is a 256 character limit) - Use JAVA_OPTS to point to a config file S3 URL, and pull that in on startup. In order for the DuraStore calls to complete quickly, and to keep the database load down, information from the DB should be cached. This could be done preemptively or lazily, but in either case, there will need to be handling for invalidation, to ensure updates to the database are picked up.	2
608	DURACLOUD-950	Beanstalk: Account determination	All calls going through DuraStore will need to determine the account for which the call should be processed. By default this should be done using the subdomain of the host used for the call. There should also be an optional query parameter added to each API call which allows specifying the accountID, as this will be necessary for testing and may be needed by other scenarios. Part of this work will be to ensure that the combination of the account and user credentials still ensures that only users with access to an account are able to make calls that affect that account.	3
609	DURACLOUD-951	Beanstalk: DuraStore calls to accounts	All calls through DuraStore in a beanstalk setup will need to pull account information then perform the expected requests on the appropriate account.	2
610	DURACLOUD-952	Beanstalk: Handling of roots	DuraStore through beanstalk will no longer have access to a single root user. Instead it will need to allow any user with the root role to have root-level access. This work is to ensure that root user access works properly. Since DuracloudUser has a root boolean property, we need to take that into account when loading user details managers from the global account repo. Currently we are relying on ROLE_ROOT associations in the account rights.	2
611	DURACLOUD-954	Users with USER role should be able to view streams on open spaces	When a space is set to open streaming (by turning streaming on via DurAdmin), any user with read access to that space should be able to view media (audio/video) that is streamed through the built-in player. Currently, this works fine for admin users, but users with only USER role just see "Unable to stream file" where the player should be. This is very likely to be caused by the fact that a task call is being made to retrieve the streaming URL, but that task call is limited to admin users. A user with access to a space should be able to make the task request to stream content from that space.	1
612	DURACLOUD-957	Backup thread runs too frequently on synctool.	Currently the synctool's backup thread is running on the same schedule as the directory polling period which is 10 seconds by default. The back up thread period should not be tied to the polling period. Instead it should run at least every 5 minutes by default. Also thread synchronization on on the ChangedList.persist method is causing threads to wait until the backup file is completely written rather than waiting only until the file list is copied in memory.	1
613	DURACLOUD-959	Beanstalk: Set up topic subscription on DuraStore init	On DuraStore init, an SNS topic subscription is set up. The SNS connection details are stored in the MC DB. When a message comes through that subscription, it picks up what changes were made and updates local cache.	3
614	DURACLOUD-961	Allow for setting chunk size in SyncTool UI	The UI version of the Sync Tool should have a setting (on the configuration tab) which allows for the size of chunks to be set.	1
615	DURACLOUD-962	SyncTool chunk size settings should be multiples of 1000	In the rest of DuraCloud, GB is defined as a multiple of 1000 (a decimal representation.) The Sync Tool currently uses a multiple of 2 (decimal representation). The Sync Tool should be updated to set the GB value to 1000^3.	1
616	DURACLOUD-965	Duradmin must use new durastore storage stats api	Duradmin must use new durastore storage stats api rather than pulling, parsing and caching results in memory. The ContentStore interface will need to be extended. Duradmin code for parsing and caching storage reports must be removed.	3
617	DURACLOUD-966	Write utility for adding storage report data to space_stats table in mill db	Currently the storage history of all spaces are stored in xml files found in x-duracloud-admin on the various instances. We want to extract this data and store it in the mill db space_stats table. To accomplish this task, we will need a tool that can convert storage reports for a single account by looping through the list of reports, extracting the data for each space and inserting a record in the space_stats table.	3
618	DURACLOUD-967	Remove storage report generation code	Remove duraboss, reporter, report client, and any dependencies on these modules.	1
619	DURACLOUD-968	Beanstalk: Connect DurAdmin to MC database and allow for calls to any account	DURACLOUD-949 and DURACLOUD-951 updated DuraStore to connect to the MC database and pull the information needed to work for all DuraCloud accounts. This work also involved caching the database information and refreshing that cache on change notificatoin. This task is to carry that work forward into a similar capability for DurAdmin.	2
620	DURACLOUD-969	SyncTool fails to connect when using a password with special characters	When a DuraCloud user sets their password to a value which includes special characters they are able to log in to the DuraCloud UI just fine, but attempting to connect using the command line SyncTool results in a stacktrace like the following: Starting up the Sync Tool ...Exception in thread "main" java.lang.RuntimeException: Could not create connection to DuraCloud (test.duracloud.org:443/durastore). Cause: invalid credentials. Check your username and password and try again. at org.duracloud.client.util.StoreClientUtil.createContentStore(StoreClientUtil.java:42) at org.duracloud.sync.SyncTool.startSyncManager(SyncTool.java:154) at org.duracloud.sync.SyncTool.runSyncTool(SyncTool.java:294) at org.duracloud.sync.SyncToolInitializer.runSyncTool(SyncToolInitializer.java:34) at org.duracloud.sync.SyncToolInitializer.main(SyncToolInitializer.java:23) at org.duracloud.syncui.SyncSelector.main(SyncSelector.java:33) An example password which provokes this issue is: H89v$2oRlaD?5Qe\YT3j. This password works fine with the SyncTool if all special characters are removed (H89v2oRlaD5QeYT3j), but fails if any remain. Once fixed, this needs to be tested in the SyncTool UI, the Retrieval Tool, as well as the chunk, stitch, and sync-optimize tools.	1
621	DURACLOUD-971	Add REST API call to retrieve latest fixity report for a given space	As discussed in DPS-123, Jim Tuttle from Duke needs a way to easily retrieve bit integrity reports generated by the DuraCloud Mill. This task is to add a new REST API call to the DuraStore API which will allow for retrieving the latest report file.	1
622	DURACLOUD-972	Remove integration-test and unit-test-db modules	The tests in the integration-test module have been disabled since 4/25/2011. The most important of these, the tests for storage provider connections, have been recreated in the "integration" module. Many of the remaining tests are likely far out of date. This task is to review the test in integration-test, move those which have real value to the integration module, then remove integration-test and its dependent unit-test-db from the codebase.	3
623	DURACLOUD-974	Finish storage report conversion: relative space byte and item count sizes.	For beanstalk version implement storage stats pie chart view of space size and space item count as of x day to match what is currently available in duracloud 3.6.0.	3
624	DURACLOUD-975	Remove dashboard from duradmin	No longer supported.	1
625	DURACLOUD-978	Bit Integrity Report REST call incorrectly returns 204 code when space is not found.	It should be returning 404	1
626	DURACLOUD-980	OpenstackProvider marker and prefix params are not url encoded on space listing calls	This problem applies to both get spaces and get spaces chunked calls. It appears that this problem was addresses for get content calls. So the OpenStackStorageProvider performs a url encode as well as some other encoding magic to get things to work properly. These issues were identified for jclouds 1.8.0 (the version we're using) in this jira: https://issues.apache.org/jira/browse/JCLOUDS-217 . According to the issue details it was resolved in jclouds 2.0.0. Short term solution is to apply the fix to the OpenStackStorageProvider. I believe the longer term fix is to upgrade to jclouds 2.0.0. That later fix will likely be somewhat involved since there were major api changes between 1.x and 2.x.	2
627	DURACLOUD-983	Durastore: TaskProviderFactory is not aware of global account repo	With the recent beanstalk-oriented changes, any durastore instance can serve requests for any account. However it appears that the TaskProviderFactoryImpl is not aware of these changes. It is still thinking that it must be initialized externally as opposed to reading from the global account store.	2
628	DURACLOUD-984	Remove classic init from DuraCloud	As we move towards an Elastic Beanstalk oriented deployment, we no longer need to support classic two phase init (ie xml posted to the application).	3
629	DURACLOUD-986	Duradmin must be aware of AccountChangeEvents	NULL	2
630	DURACLOUD-987	Mill: Handle corner case of missing storage provider	Currently the looping task producers will enter an infinite loop if a storage provider is missing. It should retry a few times. If it continues to fail it should send and an email and move on.	2
631	DURACLOUD-988	Ensure DuraCloud dependencies are version consistent and secure	This task is to add two new checks to the DuraCloud dependency chain: 1. Add the dependency convergence option to the maven enforcer (https://maven.apache.org/enforcer/enforcer-rules/dependencyConvergence.html). This compares versions in the dependency tree to ensure that only a single version of any given artifact is referenced in the build. 2. Add the enforce-victims-rule to the maven enforcer (https://github.com/victims/victims-enforcer). This checks the dependency tree against a list of dependencies known to have security vulnerabilities. After adding these requirements, there will be changes needed in the codebase to comply with any issues that are uncovered.	5
632	DURACLOUD-989	Changes to users in management console does not always trigger update in duracloud	The following scenarios appear not to cause a cache update in duracloud: 1. Removing a user from an account 2. Adding a new user to an account 3. Updating a user's rights on an account. also: double check deleting a user from all accounts works.	2
633	DURACLOUD-991	Setup Bit integrity producer to run automatically	It would be beneficial to rerun failed bit integrity runs in order to reduce the time spent managing bit integrity failures. To achieve this end we must 1. Ensure that if an instance of the producer is currently running, do not run. 2. modify the bit integrity producer to ignore any space that has run successfully within the last 60 days.	2
634	DURACLOUD-992	Looping task producers should ignore Inactive Accounts	Currently inactive accounts are not ignored.	2
635	DURACLOUD-993	500 error attempting to download file through DurAdmin	In the UNC account, the file named 20304_VT0010_0001.mp4 in the space sfc20304-open cannot be downloaded using the "Download" button. Selecting the download button results in a 500 error, which is captured in the attached "500-error-trace.txt" file. The error which is generated on the server side is captured in the attached "500-error-server-side-catalina-out.txt" file. The file in question is 4.5 GB in size. It is in a streamed space, and streaming works properly. The file can be downloaded from S3 and from DuraStore, so the issue seems to be occurring within DurAdmin.	1
636	DURACLOUD-994	Duplicate response headers from REST API	Calling the REST API for GetContentProperties return some duplicate headers, excerpted below: $ curl -I -u "${DURACLOUD_USER}:${DURACLOUD_PASSWORD}" https://${DURACLOUD_HOST}/durastore/rest-api-testing/foo2 HTTP/1.1 200 OK Date: Tue, 05 Apr 2016 19:38:14 GMT ETag: 26dc5f719443fb5ff49b87b3853d5d7c ETag: 26dc5f719443fb5ff49b87b3853d5d7c Content-MD5: 26dc5f719443fb5ff49b87b3853d5d7c Content-MD5: 26dc5f719443fb5ff49b87b3853d5d7c Last-Modified: 2016-04-05T18:36:52 Last-Modified: 2016-04-05T18:36:52	1
637	DURACLOUD-995	Beanstalk: ACLStorageProvider must be reloaded on ACL change	ACLs are currently cached in the ACLStorageProvider in order to improve performance of space reads. This works fine when a single instance of ACLStorageProvider can be assumed. However in beanstalk mode we expect that each node may have a copy of the same ACLStorageProviderCache. Changes to groups or users that occur in the management console will generate a notification to all of these instances. However, changes to the permissions made via Duradmin or the Durastore API must also cause nodes to reload their caches. One approach to resolve the issue will be to have the ACLStorageProvider generate a notification to the other nodes. It might be useful to include the source IP address in the notification so that message receivers can opt to ignore messages received from themselves.	3
638	DURACLOUD-997	Do a test run of 4.0.0-SNAPSHOT on beanstalk.	NULL	2
639	DURACLOUD-998	Create dev AMI for DuraCloud 4.x testing.	NULL	2
640	DURACLOUD-1000	Improve indexing of management console database	NULL	2
641	DURACLOUD-1001	Beanstalk: Load duracloud properties from S3	Currently -Dduracloud.config.file only supports files. In order to make it possible for us to use a plain vanilla beanstalk ami it would be better for us to load config from S3. Loading from S3 would make the instances more secure by now storing credentials on the instances themselves.	2
642	DURACLOUD-1003	root URI does not redirect to /duradmin/ in 4.0.0 on beanstalk	https://<account>.duracloud.org does not redirect to https://<account>.duracloud.org/duradmin as it does in the current production 3.x line.	2
643	DURACLOUD-1004	Remove duracloud user and password from GlobalProperties in Management Console	NULL	1
644	DURACLOUD-1005	MC: global properties form does not give useful feedback when data is invalid	NULL	1
645	DURACLOUD-1006	Investigate automate deployment to beanstalk	As a first step, it would be very helpful if commits to development version were automatically deployed to a test beanstalk environment. Additionally it would be good if integration tests were run as part of that process. NB: Since dependencies such as duracloud-db have *-SNAPSHOT versions while each duracloud instance is in development, it would be good if duracloud-db commits also trigger a rebuild of duracloud and management-console.	4
646	DURACLOUD-1007	Make MC able to read properties from S3	NULL	1
647	DURACLOUD-1008	upgrade spring security in management console	NULL	2
648	DURACLOUD-1009	upgrade spring security in snapshot	NULL	2
649	DURACLOUD-1013	Non-admin users not able to see snapshots	On a DuraCloud Vault instance (vault.d.o), I gave read/write permissions to two spaces to a non-admin user. Both of those spaces had snapshots (one complete, one in process). When logged in as the user, both spaces were visible, but no snapshots were displayed. A call to the get-snapshots REST endpoint as the user shows the two expects snapshots are listed. This is likely an issue in the UI layer.	2
650	DURACLOUD-1015	Add restful error handling to the Storage Report REST API	Currently failures are returning 500 errors.	2
651	DURACLOUD-1016	Refactor storage report API endpoints to be more consistent with the rest of the DuraCloud API	Current API endpoints: # Provides a listing of totals for content in a space over a time range /storagestats/timeseries ? spaceID & storeID & start & end # For a single day, provides a list of report details for each space in a store /storagestats/snapshot-by-day ? storeID & date Should be changed to: # Summary of a space, listed by day /report/space/{spaceID} ? storeID & startDate & endDate # Summary of a store, listed by day /report/store ? storeID & startDate & endDate # Summary of a store (for a single day), listed by space /report/store/{date} ? storeID - storeID should be optional, default to primary store Note: No changes need be made to the functionality, just to the REST endpoints.	2
652	DURACLOUD-1017	SyncTool chunking error	I have an issue chunking files larger than the max size: ERROR 2016/05/07 02:46:37 [pool-3-thread-2] (SyncWorker.java:68) [run()] - Exception syncing file /srv/perkins/duracloud/spaces/repository.duke.edu-dev/BINARIES/d4/a6/22/d4a622b5a2a64e750e3e01abaf687eceebfe06d5 was MaxChunkSize must be multiple of 1024: 1000000000 org.duracloud.common.error.DuraCloudRuntimeException: MaxChunkSize must be multiple of 1024: 1000000000 at org.duracloud.chunk.ChunkableContent.calculateBufferSize(ChunkableContent.java:93) ~[duracloudsync-3.7.3.jar:na] at org.duracloud.chunk.ChunkableContent.<init>(ChunkableContent.java:61) ~[duracloudsync-3.7.3.jar:na] at org.duracloud.chunk.ChunkableContent.<init>(ChunkableContent.java:53) ~[duracloudsync-3.7.3.jar:na] at org.duracloud.chunk.FileChunker.doAddContent(FileChunker.java:226) ~[duracloudsync-3.7.3.jar:na] at org.duracloud.chunk.FileChunker.addContent(FileChunker.java:152) ~[duracloudsync-3.7.3.jar:na] at org.duracloud.sync.endpoint.DuraStoreChunkSyncEndpoint.addUpdateContent(DuraStoreChunkSyncEndpoint.java:164) ~[duracloudsync-3.7.3.jar:na] at org.duracloud.sync.endpoint.DuraStoreSyncEndpoint.addUpdateContent(DuraStoreSyncEndpoint.java:304) ~[duracloudsync-3.7.3.jar:na] at org.duracloud.sync.endpoint.DuraStoreSyncEndpoint.doAddContent(DuraStoreSyncEndpoint.java:262) ~[duracloudsync-3.7.3.jar:na] at org.duracloud.sync.endpoint.DuraStoreSyncEndpoint.syncFileAndReturnDetailedResult(DuraStoreSyncEndpoint.java:229) ~[duracloudsync-3.7.3.jar:na] at org.duracloud.sync.mgmt.SyncWorker.run(SyncWorker.java:65) ~[duracloudsync-3.7.3.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]	1
653	DURACLOUD-1018	Management Console: Deleting a user from an account is failing	HTTP Status 403 - Invalid CSRF Token 'null' was found on the request parameter '_csrf' or header 'X-CSRF-TOKEN'. Probably is due to the fact that CSRF tokens are on by default in the new spring security.	2
654	DURACLOUD-1019	Management console: Logout link results in 404	NULL	1
655	DURACLOUD-1020	Move Bit Integrity Producer on to the Sentinel	In order to better utilize our AWS resources we should do the following: 1. Retire the Bit Integrity Producer Autoscale group and configuration. 2. Move the bit integrity producer process onto the sentinel and have it run continuously. 3. Upgrade the sentinel to an m3.medium. 4. Ensure that the bit report autoscale group is scaling down when the bit report queue has been empty for more than 3 hours.	2
656	DURACLOUD-1021	Mill: Dup Task Processor should perform delete on destination only if the source does not contain item.	In order to cover the case of content disappearing (due to provider failure) from the source provider we must make sure to delete destination store content only if the content is no longer in the source manifest. Currently if a dup task comes to dup content from a source to a dest provider, if the item doesn't exist in the primary we delete it in the destination. However the looping dup producer adds dup messages for items not in the source that are in the destination in order to synchronize any missed deletes. Thus a file that disappears from the source unexpectedly will be deleted resulting in data loss. So to fix this problem: if dup task targets a content item that does not exist in the source provider, ensure that it is not in the source manifest before deleting from destination.	2
657	DURACLOUD-1024	In the CleanUpSnapshotTask, read all items before generating delete messages	Currently the CleanupSnapshot task streams manifest items using a streaming iterator. For each item, a delete audit message is generated. Under the covers the manifest item streamer pulls items in 10K item pages. If the manifest processor in the mill begins working the list before all pages have been pulled, the manifest streamer will miss some records. To resolve, write entire list of content Ids to file before generating audit tasks.	1
658	DURACLOUD-1027	Add Chronopolis storage provider	This will be a subclass of the snapshot provider, to provide the ability to push to Chronopolis, not DPN.	5
659	DURACLOUD-1028	Update Bridge dependencies to rely on latest duracloud,spring, etc.	Update dependencies for duracloud, spring, hibernate, etc. Resolve any issues the upgrade creates.	2
660	DURACLOUD-1029	Chart data calls are returning 500 error from DuraStore	NULL	2
661	DURACLOUD-1042	Add CORS headers to Elastic Beanstalk config	We need to perform the same config on 4.x that we did on 3.x	1
662	DURACLOUD-1045	Use Manifest to Count items and Bytes for Glacier for Storage Stats	NULL	2
663	DURACLOUD-1046	Error message on Chronopolis provider says DPN	If a communication or other error occurs within the Chronopolis provider, an error dialog is shown that includes the text: "DuraCloud is not currently able to connect to DPN". This error continues to make sense for the DPN provider, but not for the Chronopolis provider. Making the error more generic, so that it could apply to either provider, would be sufficient.	1
664	DURACLOUD-1048	Tags for USER role with read-only permission are presented with no spaces	When a user with USER role is given read access to a space, they are able to see the content, but when they select the content in the UI, the list of tags are listed with no space between the tags, so they all appear as if they are a single word. Example: If the tags on a file are "test", "popcorn", and "bicycle", then they are displayed as "testpopcornbicycle"	1
665	DURACLOUD-1050	Update Duplication Monitor to work with DuraCloud 4.x	1. Remove obsolete dependencies on database 2. Retrieve space counts from storage stats rather than making a content item count by means of looping through all the content ids. 3. Ensure monitor username and password is encrypted	3
666	DURACLOUD-1051	Ensure that management console uses beanstalk settings to require https	It should work just like the duracloud elastic beanstalk.	2
667	DURACLOUD-1053	Management Console: Changing a root user's profile (password, etc) does not reset all caches	Resetting a root user's profile should reset all user data caches. Currently a USERS_CHANGED message is being sent with no accountId. The result is that instances are not registering the change.	1